Introducing Keyword-Driven Test Automation

TO PM Tutorial

10/1/2013 1:00:00 PM

"Introducing Keyword-Driven

Test Automation"

Presented by:

Hans Buwalda

LogiGear Corporation

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Hans Buwalda

LogiGear

An internationally recognized expert in testing, Hans Buwalda is a pioneer of keyword-driven

test automation, an approach now widely adopted throughout the testing industry. Originally

from the Netherlands, Hans is the CTO of LogiGear, directing the development of the successful

Action Based Testing™ methodology for keyword-driven test automation and its supporting

TestArchitect™ toolset. Prior to joining LogiGear, Hans served as project director at CMG (now

CFI).

8/20/2013

1

© 2013 LogiGear Corporation. All Rights Reserved

Hans Buwalda

LogiGear

STAREAST 2013, Tutorial TO

Orlando, Tuesday April 30

Introducing

Keyword-driven

Test Automation


Introduction

industries

roles in testing

8/20/2013

2


About LogiGear

Software testing company, around since 1994

Testing and test automation expertise, services and

tooling consultancy, training

test development and automation services

"test integrated" development services

Aims to be thought leader, in particular for large and

complex test projects

Products: TestArchitect™, TestArchitect for Visual Studio™

integrating test development with test management and automation

based on modularized keyword-driven testing

www.logigear.com

www.testarchitect.com


About Hans

Dutch guy, living and working in California since 2001, as

CTO of LogiGear

Background in math, computer science, management

Original career in management consultancy, since 1994

focusing on testing and test automation keywords, agile testing, big testing, . . .

www.happytester.com

hans @ logigear.com

8/20/2013

3


Topics for this tutorial

Introduction to keyword driven testing including "Action Based Testing", my own flavor of it...

Comparison to other techniques for automation

Recommendations for a successful application of keyword

driven testing test design

automation

organization

Some ideas for specific situations: data driven testing

non-ui testing

multi-media

protocols

initial data

Not everything will be equally interesting, or accessible, to

everybody


specification development test

Testing Under Pressure

DEADLINE

8/20/2013

4


specification development test

Develop tests in time:

• Test design

• Auditing, acceptance

• Preparations

• Automation

Testing Under Pressure

DEADLINE


The 5% Rules of Test Automation

No more than 5% of all test cases should be

executed manually

No more than 5% of all efforts around

testing should involve automating the tests

8/20/2013

5


Why a High Automation Degree ??

The best way to prepare for efficiency in the crunch zone good manual test cases can help too, but marginally

Buy time to do more "exploratory testing", and better test

development

Credible pay-off for the cost of introduction of automation initial costs are: tooling, learning curve, adaptation of existing tests

Automation is better positioned to identify “bonus bugs” on average 15% of fixes cause new bugs

many of these bugs are hard to find without integral testing • often a result of violating overall architectures

• the bugs occur because data is left in inconsistent state

Automated tests have a better chance of being kept up to date if they

form the majority of the testware

Automation can be re-run, for example as part of the continuous

integration process either specific based on code changes, or integral, to also catch bonus bugs


Why < 5% Automation Efforts ??

Automation should not dominate testing it is not a goal in itself

may never be a bottleneck

Testers should be able to focus on testing better tests (higher ambition level)

communication with stake holders

High automation efforts can aggravate the

“crunch zone”, instead of relieving it “invitation to Murphy’s law”

automation should deliver, not dominate…

8/20/2013

6


Record and Playback

select window "Logon"

enter text "username", "administrator"

enter text "password", "testonly"

push button "Ok"

select window "Main"

push button "New Customer"

expect window "Customer Information"

select field "First Name"

type "Paul"

select field "Last Name"

type "Jones"

select field "Address"

type "54321 Space Drive"

.

.

.


Scripting

Test Case

Design

Test Case

Automation

Test Case

Execution

TEST DESIGNER

AUTOMATION ENGINEER

MR. PLAYBACK

8/20/2013

7


Example scripting

/// <summary> /// AddItems - Use 'AddItemsParams' to pass parameters into this method. /// </summary> public void AddItems() { #region Variable Declarations WinControl uICalculatorDialog = this.UICalculatorWindow.UICalculatorDialog; WinEdit uIItemEdit = this.UICalculatorWindow.UIItemWindow.UIItemEdit; #endregion // Type '{NumPad7}' in 'Calculator' Dialog Keyboard.SendKeys(uICalculatorDialog, this.AddItemsParams.UICalculatorDialogSendKeys, ModifierKeys.None); // Type '{Add}{NumPad2}{Enter}' in 'Unknown Name' text box Keyboard.SendKeys(uIItemEdit, this.AddItemsParams.UIItemEditSendKeys, ModifierKeys.None); }

State of the art, but stuff for coders . . .


Common in automation tools nowadays (but with different styles)

Identify tasks for both test development and automation

The test developer creates tests using actions

Each action consists of a keyword and arguments

The automation task focuses on automating the actions

Each action is automated only once

Keywords, Action Words

number name quantity

new product P-9009 Sledge Hammer 5

number quantity

add quantity P-9009 20



number quantity

check quantity P-9009 34

actions, each with a

keyword and

arguments

"34" is the

expected value

here

read from top

to bottom

fragment from a test with actions

8/20/2013

8


Potential benefits of keywords

More tests, better tests more breadth

more depth

Fast, results can be quickly available the design directly drives the automation

Separates the tests from the technical scripting language easier to involve business subject matter experts

the action format allows for easy readability

Less efforts for automation "script free" in most cases

Automation more stable and maintainable limited and manageable impact of changes in the system under test

Develop tests more early in the life cycle deal with execution details later

. . .


Risks of keyword approaches

Often seen as silver bullet, complications are

underestimated often treated as a technical "trick"

testers can get squeezed and marginalized • developers and users dictating tests

• automation engineers dictating actions

or testers get the automation responsibility, thus becoming pseudo

programmers

The method needs understanding and experience to be

successful pitfalls are many, and can have a negative effect on the outcome

Lack of method and structure can risk manageability maintainability not as good as hoped

results can be disappointing, approach will be blamed

8/20/2013

9


Combining Approaches . . .

Use keywords for the automation-ready

description of test cases

Use scripting to set up structured automation for

the actions

Use record and playback to record keywords


Enter a user id that is greater than 10 characters,

enter proper information for all other fields,

and click on the "Continue" button

There should be an error message stating

that "User Id must be less than 10

characters".

Enter a User Id with special character's), enter

proper information for all other fields

and click on the "Continue" button

An error message should be displayed

indicating that "User Id cannot contain

some special characters".

Enter the information, with a password of 4

characters and click on the "Continue" button

Check for an error message saying:

"Password must contain at least 5

characters".

Comparing Formats

Most values are implicit. The tester has to

figure them out during execution…. classic format

user id message

check registration dialog aaaaabbbbbc User Id must be less than 10 characters

user id message

check registration dialog résoudre User Id cannot contain some special characters

password message

check registration dialog test Password must contain at least 5 characters

keywords

Execution instructions are

repeated in multiple test cases

Welcome to . . . "Complete Test Make-Over"

. .

8/20/2013

10


Keywords is not just test automation

Can also be used for other than testing: data entry chores

training purposes

Can also be used for manual testing for example with a manual testing dialog

even can show instructions, with placeholders for values

Action login <user> <password> Enter "<user>" in the user name field, and "<password>" in the password field.

Test line user name password

login hansb starwest

What the manual tester would see Enter "hansb" in the user name field, and "starwest" in the password field.


Keywords need a method

By themselves keywords don't provide much scalability they can even backfire and make automation more cumbersome

a method can help tell you which keywords to use when, and how to

organize the process

Today we'll look at Action Based Testing (ABT) addresses test management, test development and automation

large focus on test design as the main driver for automation success

Central deliveries in ABT are the "Test Modules" developed in spreadsheets

each test module contains "test objectives" and "test cases"

each test module is a separate (mini) project, each test module can

involve different stake holders

8/20/2013

11


Don't just automate manual testing



8/20/2013

12



Good automated testing is not the same as automating good manual testing. . .


Action Based Testing (ABT)

Uses the keyword format as a basis for a method covers test management, test development and automation

with a large focus on test design as the main driver for automation

success

method is specific, but concepts are generic

The central product in ABT is the "Test Module", not the

test case like chapters in a book

test cases are part of the test modules, they are typically the result

(rather than the input) of test development

test development is seen as having both analytical and creative

aspects

developed as spread sheets, external from the automation, with a

well defined flow

easier to manage: each test module is a separate (mini) project, each

test module can involve different stake holders

8/20/2013

13


Action Based Testing

Test Development Plan

Test Cases

Test

Objectives

Test Module 1

Test Cases

Test

Objectives

Test Module 2

Test Cases

Test

Objectives

Test Module N

Actions

. . .

ACTION AUTOMATION

Break down

Automate


Example of business level test module

Consists of an (1) initial part, (2) test cases and (3) a final part

Focus is on business functionality, with a clear business scope

Navigation details are avoided

TEST MODULE Car Rental Payments

user

start system john

TEST CASE TC 01 Rent some cars

first name last name car

rent car John Doe Ford Escape

rent car John Doe Chevvy Volt

last name amount

check payment Doe 140.4

FINAL

close application

8/20/2013

14


Example of an interaction level test module

Lay-out the same, with an initial part, test cases and a final part

Interaction details that are the target of the test are not hidden

Focus is not on business ("is the payment amount correct"), but on

interaction ("do I see the payment amount")

TEST MODULE Screen Flow

user

start system john

TEST CASE TC 01 Order button

window button

click main create order

window

check window exists new order

FINAL

close application


Variables and expressions with keywords

This test does not need an absolute number for the

available cars, just wants to see if a stock is updated

As a convention we denote an assignment with ">>"

The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car available

get quantity Chevvy Volt >> volts




car expected

check quantity Chevvy Volt # volts - 2

8/20/2013

15


Data driven testing with keywords

The test lines will be repeated for each row in the data set

The values represented by "car", "first" and "last" come

from the selected row of the data set

DATA SET cars

car first last value

Chevvy Volt John Doe 40000

Ford Escape Mary Kane 22500

Chrysler 300 Jane Collins 29000

Buick Verano Tom Anderson 23000

BMW 750 Henry Smyth 87000

Toyota Corolla Vivian Major 16000

TEST CASE TC 03 Check stocks

data set

use data set /cars

car available

get quantity # car >> quantity


rent car # first # last # car

car expected

check quantity # car # quantity - 1

repeat for data set


Automating keyword tests

Function Interpret

While not end of test

Read next line

Split the line into arguments

Look up the keyword in the "action list"

Execute the function belonging to the keyword

Report the results of this line

Repeat for next line

End

Report Interpreter

Keywords are useful, but technical not complex. It is not hard to make a

simple keyword interpreter. Many test tools also have keyword options in

some form or another.

8/20/2013

16


Automation: example test lines

System Under Test Test Lines

key

click key one

click key plus

click key one

click key equals

expected

check display 2.


Create an "action function" for each action

identify the

UI element

perform the

operation

identify the

UI element

perform the

check

# map an action to its function

def divertCalculator(action):

if action == "click key": action_ClickKey()

elif action == "check display": action_CheckDisplay()

. . .

else: Error("Don't know action: " + action)

. . .

# action "click key", click a key on the calculator

def action_ClickKey():

keyName = Argument("key")

key = OpenElement("calculator", Argument("key"))

key.click()

# action "check display", verify the value of the display

def action_CheckDisplay():

display = OpenElement("calculator", "display")

display.Check(Argument("expected"))

Code module: "mod_Calculator"

get the value for

argument "key"

from the test line

8/20/2013

17


Example: script for an action "check sort order"

# get table object, column number and column count

windowName = LIBRARY.NamedArgument("window")

tableName = LIBRARY.NamedArgument("table")

columnName = LIBRARY.NamedArgument("column")

table = ABT.OpenElement(windowName, tableName)

column = table.GetColumnIndex(columnName)

rowCount = table.GetRowCount()

# check the sort order, row by row

previous = table.GetCellText(0, column)

for i in range(1, rowCount):

current = table.GetCellText(i, column)

if current < previous :

LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)

return

previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

The following action script will verify whether the rows in a table are sorted:

find the table in the UI

if a value is smaller than before, fail the test

if all rows are ascending, pass the test

get arguments from the test line

def action_checkSortOrder():


Example application

For try-outs and demonstrations only

Application is made in WPF

8/20/2013

18


A Test Module for the application

We click a tree node, and then do a check

The actions here are built-in in the framework


Making a new "action"

This action definition uses existing actions to create a new action called "check bitrate"

Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators

create a node path

from the first two

arguments the expected value is

given by the 3rd argument

the arguments of

the new action

name of the new action

8/20/2013

19


Using the action in a test

These test lines don't care about the navigation in the UI of the application, the focus is functional: verify data

Such functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term


4000

tests

250

actions

2000

tests

22 functions

200

actions 20 functions

In a good application of the keywords approach a large increase in test cases (like doubling

the amount) should result in a modest increase in actions, and a minor increase, if any, in

programmed action functions.

Do the count down . . .

8/20/2013

20


Identifying controls

Identify windows and controls, and assign names to them

These names encapsulate the properties that the tool can use to identify the windows and controls when executing the tests


Mapping the interface

An interface mapping (common in test tools) will map windows and

controls to names

When the interface of an application changes, you only have to update

this in one place

The interface mapping is a key step in your automation success, allocate

time to design it well

INTERFACE ENTITY library

interface entity setting title {.*Music Library}

ta name ta class label

interface element title text Title:

interface element artist text Artist:

interface element file size text File size (Kb):

ta name ta class position

interface element playing time text textbox 4

interface element file type text textbox 5

interface element bitrate text textbox 6

ta name ta class position

interface element music treeview treeview 1

8/20/2013

21


Some Tips to Get Stable Automation

Make the system under test automation-friendly

Use "active" timing

Test your automation

Use automation to identify differences between

versions of the system under test (in particular

the interfaces)

Keep an eye on the test design


Look for properties a human user can't see, but a test tool can

This approach can lead to speedier and more stable automation interface mapping is often bottleneck, and source of maintenance problems

with predefined identifying property values in interface map can be created without "spy" tools

not sensitive to changes in the system under test

not sensitive to languages and localizations

Examples: "id" attribute for HTML elements

"name" field for Java controls

"AccessibleName" property in .Net controls (see below)

Automation-friendly design: hidden properties

8/20/2013

22


Mapping the interface

Instead of positions or language dependent labels, an internal property

"automation id" has been used

The interface definition will be less dependent on modifications in the UI

of the application under test

If the information can be agreed upon with the developers, for example in

an agile team, it can be entered (or pasted) manually and early on

INTERFACE ENTITY library

interface entity setting automation id MusicLibraryWindow

ta name ta class automation id

interface element title text TitleTextBox

interface element artist text SongArtistTextBox

interface element file size text SizeTextBox

interface element playing time text TimeTextBox

interface element file type text TypeTextBox

interface element bitrate text BitrateTextBox

ta name ta class automation id

interface element music treeview MusicTreeView


Active Timing

Passive timing wait a set amount of time

in large scale testing, try to avoid passive timing altogether: • if wait too short, test will be interrupted

• if wait too long, time is wasted

Active timing wait for a measurable event

usually the wait is up to a, generous, maximum time

common example: wait for a window or control to appear (usually the test tool will do

this for you)

Even if not obvious, find something to wait for...

Involve developers if needed relatively easy in an agile team, but also in traditional projects, give this priority

If using a waiting loop make sure to use a "sleep" function in each cycle that frees up the processor (giving the

AUT time to respond)

wait for an end time, rather then a set amount of cycles

8/20/2013

23


Things to wait for...

Wait for a last control or elements to load developers can help knowing which one that is

Non-UI criteria API function

existence of a file

Criteria added in development specifically for this purpose, like: "disabling" big slow controls (like lists or trees) until they're done loading

API functions or UI window or control properties

Use a "delta" approach: every wait cycle, test if there was a change; if no change, assume that the

loading time is over:

examples of changes: • the controls on a window

• count of items in a list

• size a file (like a log file)


Test Design

Effective test breakdown (into test modules) make sure every test module has a clear focus

keep different kinds and levels of tests separate

Right level of actions as “high level” if possible, hiding as many details as

much as possible

but not if the details are relevant for the test

It is my believe that successful automation is not a technical

challenge. It is most of all a test design challenge.

8/20/2013

24


The Three “Holy Grails” of Test Design

Metaphor to depict three main steps in test design

Using "grail" to illustrate that there is no one perfect

solution, but that it matters to pay attention (to search)

About quality of tests, but most of all about scalability and

maintainability in BIG projects

Right approach for each test module

Proper level of detail in the test specification

Organization of tests into test modules


What's the trick...

8/20/2013

25


What's the trick...

Have or acquire facilities to store and organize

you content

Edit your stuff

Decide where to put what assign and label the shelves

Put it there

If the organization is not sufficient anymore, add

to it or change it


Breakdown Criteria

Straightforward Criteria Business tests versus interaction tests

Architecture of the system under test (client, server, protocol, sub

systems, components, modules, ...)

Functionality (customers, finances, management information, ...)

Kind of test (navigation flow, negative tests, response time, ...)

Ambition level (smoke test, regression, aggressive, …)

Additional Criteria Stakeholders (like "Accounting", "Compliance", "HR", ...)

Complexity of the test (put complex tests in separate modules)

Technical aspects of execution (special hardware, multi-station, ...)

Overall project planning (availability of information, timelines, sprints, ...)

Risks involved (extra test modules for high risk areas)

8/20/2013

26


Properties of a good Breakdown

Reflects the level of tests

Well differentiated and clear in scope

Balanced in size and amount

Modules mutually independent

Fitting the priorities and planning of the project


Example breakdown

Tests of user interface does function key F4 work

does listbox xyz the right values

is the tab order correct

Form Tests, do all the forms (dialogs, screens, pages) work: can data be entered and is it stored well

is displayed data correct

split these from everything else

Function tests, do individual functions work can I count the orders

Alternate paths in use cases can I cancel a transaction

End-to-end tests do all components of a system work well together in implementing the business processes

like enter sale order, then check inventory and accounting

Tests with specific automation needs like multi station tests

Tests of non-UI functions

High ambition tests (aggressive tests) can I break the system under test

8/20/2013

27


Example of an application under test

Various item types tests

actions

interface definitions

data sets

folders

...

Various operations open

cut, copy, paste

check out

...

Various ways to initiate an operation context menu, with or without accelerator key

main menu, with or without accelerator key

toolbar

short cut key

function key

drag and drop

double click

....


Defining some modules

Test modules for operations primary and alternate paths

various values for fields like "comment" in check-in

paste in other projects

copy and paste various groups

not necessarily on each item type

Test modules for items address all item types at least once

on each item type perform each operation

not necessarily each variant of each operation

UI handling try for UI command if it starts the intended operation

not necessarily on each item type or operation variant

Concurrency try concurrency sensitive operations with multiple stations

in varying concurrency scenarios, with and without local "refreshes"

not necessarily each item type or operation variant

certainly not each UI command included

8/20/2013

28


What I have seen not work

"Over-Checking": having checks (for example based on

navigation) that do not fit the scope of the test

Forcing data driven: make all tests data driven (variables,

data files) without clear reason

Combinatorial explosions: test all ... for all ... in all ...

All actions high level (or all actions low level)

Splitting the process in "test designers" and "test

implementers" it's ok for one party to define tests for another party, but let him/her

focus on "what" and not "how"

One-directional focus on the forms in an application think about data, transactions, states, ...


"Thou Shall Not Debug Tests..."

Large and complex test projects can be hard to "get to

run"

If they are however, start with taking a good look again at

your test design...

Rule of thumb: don't debug tests. If tests don't run

smoothly, make sure: lower level tests have been successfully executed first -> UI flow in

the AUT is stable

actions and interface definitions have been tested sufficiently with

their own test modules -> automation can be trusted

are you test modules not too long and complex?

8/20/2013

29


What about existing tests?

Compare to moving house: some effort can't be avoided

decide where to put what, then put it there

consider a moving company to help

Adopt the module model define the modules, and their scope

worry about the existing test cases later

Moving considerations be selective, moving is a chance, unlikely you get that

opportunity again

for the important modules: design as normal but harvest

from existing set

avoid porting over test cases "step by step", in particular

avoid over-checking


Grail 2: Approach per Test Module

Organize the test design process around the test modules

Plan the test module: when to develop: is enough specification available

when to execute: make sure the functionality at action level is well-

tested and working already

Process: analysis of requirements

formulation of "test objectives"

create "test cases"

Identify stakeholders and their involvement: users, subject matter experts

developers

auditors

Choose testing techniques if applicable: boundary analysis, decision tables, transition diagrams, soap opera

testing, ...

8/20/2013

30


Eye on the ball, Scope

Always know the scope of the test module

The scope should be unambiguous

The scope determines many things: what the test objectives are

which test cases to expect

what level of actions to use

what the checks are about and which events should

generate a warning or error (if a “lower” functionality

is wrong)


State your Objectives . . .

...

TO-3.51 The exit date must be after the entry date

...

test objective TO-3.51

name entry date exit date

enter employment Bill Goodfellow 2002-10-02 2002-10-01

check error message The exit date must be after the entry date.

8/20/2013

31


Grail 3: Specification Level, choosing actions

Scope of the test determines the specification level

As high level as appropriate, as little arguments as

possible Use default values for non-relevant arguments

Clear names (usually verb + noun usually works well) to standardize action names: standardize both the verbs and the nouns, so

"check customer" versus "verify client" (or vice versa)

tests are not C++ code: avoid "technical habits", like mixed case and (worse)

underlines

Manage the Actions

Document the Actions

By-product of the test design


Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function

• the "#" means an expression, in this case a variable

• the ">>" assign to a variable for use later on in the test

key

key navigate F7

key navigate 3

page tab

locate page tab Scan Criteria

w indow

wait for controls loaded search

text

check breadcrumb general functions > search

w indow control value

select search scan direction Backward

w indow control value

enter value search business date match # bus date

source control

click search go

w indow

wait for controls loaded search results

w indow control variable

get search results sequence number >> seq num

8/20/2013

32


variable

get sequence number >> seq num

Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function

• the "#" means an expression, in this case a variable

• the ">>" assign to a variable for use later on in the test


Environments, configurations

Many factors can influence details of automation language, localization

hardware

version of the system under test

system components, like OS or browser

Test design can reflect these certain test modules are more general

others are specific, for example for a language

But for tests that do not care about the

differences, the automation just needs to "deal"

with them shield them from the tests

8/20/2013

33


Capture variations of the system under test in the actions and interface

definitions, rather than in the tests (unless relevant there).

Can be a feature in a test playback tool, or something you do with a global

variable or setting.

Variation Variation Variation

"Variations"

"Master Switch"

Actions, Interface Definitions

. . .


Possible set up of variations

linked variation

keyworded variation

Specify for example in a dialog when you start an execution:

8/20/2013

34


Non-UI Testing

Examples application programming interfaces (API’s)

embedded software

protocols

files, batches

databases

command line interfaces (CLI’s)

multi-media

mobile devices

Impact is mainly on the automation test design should in most cases be transparent towards the specific

interfaces

Often non-UI automation can speed up functional tests

that do not address the UI


Multiple System Access

System (part)

Under Test

Automation Scheme

API

access

protocol

access

UI

access

database

access

Test Modules, driving either

one or multiple interfaces

8/20/2013

35


Device Testing

Software Under Test

Agent ABT

Automation

Interface

Info

Testing Host Device

Android


Multimedia: The "Play List" Approach

Approach applicable for graphics, videos, sound

fragments, etc

The test includes "questions": what the tester should see or hear

like "Are the matching areas blue?"

actions like "check picture"

The test tool keeps a "play list" during the run items are captured and stored

after the run, the tester is presented with the items,

and the matching questions

the tester acknowledges/falsifies

the system stores those passed items

if during the next run the items are the same as

earlier passed ones, the tester is not asked again

8/20/2013

36


Performance Testing

The topic is complex, but to create tests can be quite

straightforward actions like "generate load <how much>" and "check response time <max

wait>"

use one tool to generate load (like JMeter), another to run the "normal"

functional test

Often performance testing isn't testing, but more close to

research analysis bottle necks and hot spots (for example discontinuities in response

times, means buffers are full)

application of statistical techniques like queuing theory

how to realistically mimic large scale productions situations in smaller test

environments

The three controls you can/should address: hardware (equipment, infrastructure, data centers, etc)

software (programs, database models, settings, etc)

demands (1 second may cost 10 times more than 2 seconds)

See also: "Load Testing for Dummies", Scott Barber, gomez.com


Organization

Much of the success is gained or lost in how you organize the

process part of the teams

who does test design

who does automation

what to outsource, what to keep in-house

Write a plan of approach for the test development and automation scope, assumptions, risks, planning

methods, best practices

tools, technologies, architecture

stake holders, including roles and processes for input and approvals

team

. . .

Assemble the right resources testers, lead testers

automation engineer(s)

managers, ambassadors, ...

Test design is a skill . . . Automation is a skill . . . Management is a skill . . .

. . . and those skills are different . . .

8/20/2013

37


Life Cycles

Product life cycles, rather than task life cycles

The project planning and execution largely determines when the

products are created

system

development

test

development

test

automation


Typical Time Allocation

TEST DEVELOPMENT

AUTOMATION

time

eff

ort

s

8/20/2013

38


Keywords and ABT in Agile

Keywords are suitable for agile projects: tests are easier to create and understand, in particular for non-programmers

they allow test development without a need for details that haven't been

defined yet

automated tests can quickly follow changes in the system under test

Action Based Testing in itself is quite agile focused on products and cooperation

flexible in process, in fact each test module can have its own process

test modules are usually very suitable to drive system development

However, ABT relies on high level test design for best

results identifying test modules

in larger scale projects this may require at least some overall test planning

activities that are not necessarily easy to do in a single scrum team


Test Development and Automation in sprints

Test Module

Definition

(optional)

Test Module Development

Interface Definition

Action Automation

Test Execution

Sprint Products Product Backlog

Test re-use

Automation re-use

product owner

team prod owner

& team

User stories

Documentation

Domain understanding

Acceptance Criteria

PO Questions

Situations

Relations

Agile life cycle

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules

8/20/2013

39


Test automation in sprints

Try keep the main test modules at a similar level as the

user stories and acceptance criteria test modules can double as modeling device for the sprint

Aim for "sprint + zero", meaning: try to get test

development and automation "done" in the same sprint,

not the next one next one means work clutters up, part of team is not working on the

same sprint, work is done double (manually and automated), ...

Make sure you can do the interface mapping by hand

(using developer provided identifications) can do earlier, before UI is finalized, and

recording of actions (not tests) will go better

Also plan for additional test modules: low-level testing of the interaction with the system under test (like

UI's)

crossing over to other parts of the system under test


Fitting in sprints

Agree on the approach: questions like does "done" include tests developed and automated?

do we see testing and automation as distinguishable tasks and

skillsets

is testability a requirement for the software

Create good starting conditions for a sprint: automation technology available (like hooks, calling functions, etc)

how to deal with data and environments

understanding of subject matter, testing, automation, etc

Make testing and automation part of the evaluations

Address tests and automation also in hardening sprints

Just like for development, use discussions with the team

and product owners to deepen understanding: also to help identify negative, alternate and unexpected situations

8/20/2013

40


Summary

Keywords is one of the techniques for automated

testing, in addition to record & playback and

scripting

In itself keywords are not a silver bullet, it needs

a good approach, careful planning and good

organization to be successful

Keywords can work for GUI testing, but equally

well for a variety of other purposes


Some References

1. Testing Computer Software, Cem Kaner, Hung Nguyen, Jack Falk, Wiley

2. Lessons Learned in Software Testing, Cem Kaner, James Bach, Bret Pettichord, Wiley

3. Experiences of Test Automation, Dorothy Graham, Mark Fewster, Addison Wesley, 2012

4. Automating Software Testing, Dorothy Graham, Mark Fewster, Addison Wesley

5. Action Based Testing (overview article), Hans Buwalda, Better Software, March 2011

6. Action Figures (on model-based testing), Hans Buwalda, Better Software, March 2003

7. Integrated Test Design & Automation, Hans Buwalda, Dennis Janssen and Iris Pinkster, Addison Wesley

8. Soap Opera Testing (article), Hans Buwalda, Better Software Magazine, February 2005

9. Testing with Action Words, Abandoning Record and Playback, Hans Buwalda, Eurostar 1996

10. QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September 2006

11. The 5% Solutions, Hans Buwalda, Software Test & Performance Magazine, September 2006

12. Happy About Global Software Test Automation, Hung Nguyen, Michael Hackett, e.a., Happy About

Technology

Introducing Keyword-Driven Test Automation