Delivering Test Automation Success

8/14/2019 Delivering Test Automation Success

1/136

Managed TestAutomation

March 31 April 2, 2009

Hans Buwa lda , CTO, Log iGear Corpora tion

"Managing Mr. Playback . . ."


2/136

Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Introduction

Who is here? industries?

involvement in testing?"Mr. Playback"

wants to know...


3/136


Implement effective and maintainable automated tests

Managing testing and test automation

Main focus: concepts, techniques and experiences To a lesser extend: in-depth treatments, full coverage of possible issues

We will use the "Action Based Testing" method as aguideline today, as an example of an approach: Representative of keyword-based approaches Incorporates most of the key ideas and concepts

Home match for your teacher

Not everything will be equally interesting, or accessible, toeverybody

Objectives


4/136


Outline of this Tutorial

TESTAUTOMATION

TEST

DEVELOPMENT

ORGANIZATION&

MANAGEMENT

INTRODUCTIONAND

OVERVIEW

1

2 3

4


5/136


Introduction and Overview


6/136


specification development

Testing Under Pressure

DEADLINE

testing


7/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

specification development testing

Develop tests in time:

Test design Auditing, acceptance Preparations

Automation

Testing Under Pressure

DEADLINE



The Challenges for a Test Process

Testing should be effective

Testing should be efficient

Testing should be under control

Testing should be fun



Organize it Well

Test Development

Test Automation

Technology/Infrastructure

ProductionMarketing/

Sales

SystemDevelopment

End UserDepartments

Quality Assurance

Management

After Sales/Help Desk

Customers

Vendors

Government

Agencies

Publicity

EXTERNAL INTERNAL


10/136



Why test?

People expect us to do

Somebody wants us to

Increases certainty and control Showing absence of problems

Finds faults, saving time, money, damage Showing presence of problems


12/136



Why Automate Tests?

It is more fun

Can save time and money potentially improving time-to-market

Reduces involvement of valuable specialists "cans" application business and application knowledge

Consolidates a structured way of working when established as integral part of system development process

Speeds up development shorter turnaround between changes

more security



Why not Automate?

Can rule out the human elements promotes "mechanical" testing might not find "unexpected" problems

More sensitive to good practices pitfalls are plentiful

Creates more software to manage

Needs/uses technical expertise in the test team

Tends to dominate the testing process at the cost of good test development



Different Points of View

1. Does the system comply with the requirementsthe positive angle

2. Are there any problems we should know about

the negative angle

3. Will the system work in practicethe pragmatic angle



What are Typical Problems

No clear direction what, when, how, who, ...

Focus on tools and technology

Test design is not well thought through main driver for success of failure

Automation lacks architecture, is not transparent and hardto manage

Testing is underestimated or avoided testing is difficult and expensive management is often in denial it looks unattractive to spend money on testing

Test automation is underestimated test automation can be very difficult "use the head first, before using the hands"



The 5% Challenge for Test Automation

No more than 5% of all test cases should be

executed manually

No more than 5% of all efforts around

testing should involve automating the tests



Approaches to Automate Test Execution

Record & Playback

Scripting

Keywords

see also: "Software Test Automation", Mark Fewster and Dorothy Graham



1. Record and Playback

select window "Logon"

enter text "username", "administrator"

enter text "password", "testonly"

push button "Ok"select window "Main"

push button "New Customer"

expect window "Customer Information"

select field "First Name"type "Paul"

select field "Last Name"

type "Jones"

select field "Address"type "54321 Space Drive"

.

.

.



Can help well to generate early versions ofindividual steps in a larger automated script

Also useful to get to know a test tool, or the UI ofthe system under test

In its pure form maintainability of generated testscripts can become a major liability

Puts the testers in the automation role "loose a good tester, gain a poor programmer"

1. Record and Playback



2. Scripting

Test CaseDesign

Test CaseAutomation

Test CaseExecution

TEST DESIGNER

AUTOMATION ENGINEER

MR. PLAYBACK



2. Scripting

. . .Function EnterCustomer ( FirstName, LastName, Address )

Click("New Customer");ExpectWindow("New Customer");EnterField("First Name", FirstName);

EnterField("Last Name", LastName);EnterField("Address", Address);. . .Click("OK");

End Function

. . .

Function MainLogon();EnterCustomer("Mary", "Jones", "123 Palm Drive");EnterCustomer("Paul", "Franklin", "321 Regent Street");. . .LogOff();

End Function. . .



2. Scripting

Structured process test cases are designed first and then automated automation is managed as a programming task

done by a specialized automation engineer

Engineering approach emphasis on structure and re-use similar to regular programming projects common tasks (like "logon") are factored out data can be made external to the programming

High quality, limited coverage approach is good (see above), but is essentially a waterfall approach efforts are substantial, burdening engineering capacity hard to keep manageable, in particular in large projects sensitive to changes in the behavior of the application under test



3. Keywords, Action Based Testing

number name quantitynew product P-9009 Sledge Hammer 5

number quantity

add quantity P-9009 20

add quantity P-9009 3add quantity P-9009 6

number quantity

check uantit P-9009 34

add quantity

check quantity

new product

function action_AddQuantity. . . .

end

function action_CheckQuantity. . . .

end

function action_NewProduct. . . .

end




Describe test cases as a series of actions withkeywords and arguments, in a sheet like format

Externalize those descriptions out of the scripting

Let the testers (not the automation engineer)

define the keywords, as by-product of the testdesign

Organize the test cases into "test modules", eachwith a clear and differentiated scope

Focus the automation efforts on the actions, not

on the test cases


26/136



Test Development Plan

Test Cases

TestObjectives

Test Module 1

Test Cases

TestObjectives

Test Module 2

Test Cases

TestObjectives

Test Module N

Actions

. . .

ACTION AUTOMATION

Break down

Automate


27/136


Example of a Test Module, "high level"

test objectivetest objective


28/136


Example of a Test Module, "low level"

TEST MODULE Product Maintenance

VERSION 1.0

. . .

TEST CASE Test Case 03 Use an invalid number

test requirement add 003 A number must be divisable by 9

source control destinationclick main new new product

window control value

enter new product number P9001

source control

click new product process

text

check message Invalid product number.

test objective


29/136


Enter a user id that is greater than 10 characters,enter proper information for all other fields,and click on the "Continue" button

There should be an error message statingthat "User Id must be less than 10characters".

Enter a User Id with special character's), enterproper information for all other fieldsand click on the "Continue" button

An error message should be displayedindicating that "User Id cannot containsome special characters".

Enter the information, with a password of 4

characters and click on the "Continue" button

Check for an error message saying:

"Password must contain at least 5characters".

Comparing FormatsMost values are implicit. The tester has tofigure them out during execution.

classic format

user id message

check registration dialog aaaaabbbbbc User Id must be less than 10 characters

user id message

check registration dialog rsoudre User Id cannot contain some special characters

password message

check registration dialog test Password must contain at least 5 characters

keywords

Execution instructions arerepeated in multiple test cases


30/136


Common Terminology

"Keywords" usually base level, interacting with the user interface

related to control classes (like textboxes, lists and menu's)

examples: "click", "enter", "select"

"Business Templates" typically high level, making automation scripts re-usable

examples: "login", "enter transaction"

"Actions" can be at multiple levels (interface, business oriented, test logic, ...) defined as part of the test design (not per se to re-use existing

automation)

a method like Action Based Testing allows earlier defined actions to

be used to define higher level action words


31/136


32/136


Risks of the Keyword Methods

Sometimes hailed as panacea, complications can beunderestimated often treated as a technical "trick"

The method needs understanding and experience to besuccessful pitfalls are many, and can have a negative effect on the outcome

Lack of structure can lead to lack in manageability in particular in the test design

Automation might take more time and costs than expected

Impact of changes might be higher than expected

Results can be disappointing and boring


33/136


Combining Approaches . . .

Use keywords for the automation-readydescription of test cases

Use scripting to set up structured automation forthe actions

Use record and playback to record actions


34/136


Some "Paradigm Shifts" . . .

From testing to: test development

From process oriented to: product oriented

From being part of a waterfall to: agile testdevelopment

From automation-centric to: test-centric

From selective automation to: integral automation

From requirement-based to: scenario-based


35/136


Test Development

TESTAUTOMATION

TESTDEVELOPMENT

ORGANIZATION& MANAGEMENT


36/136


Recent Software Error

Six months before Global Surveyor fellsilent, engineers sent up incorrectsoftware commands that disabled its

solar panels . . .

source: www.cnn.com/2007/TECH/space/04/13/mars.surveyor.ap/index.html


37/136


...TO-3.51 The exit date must be after the entry date...

State your Objectives . . .

// Test Case: Billy Goodfellow// Test Objective: TO-3.51Function EnterBilly

Press "New Employee"

Enter Field "Name", "Goodfellow"Enter Field "First Name", "Billy"Enter Field "Entry Date", "2002-10-02"Enter Field "Exit Date", "2002-10-01". . .

Press "Process"Find Text "Exit date must be after entry date"if Found then

Result passedelse

Result failed, "No error message"

. . .End Function


38/136


State your Objectives . . .

...TO-3.51 The exit date must be after the entry date...

test objective TO-3.51

name entry date exit date

enter employment Bill Goodfellow 2002-10-02 2002-10-01

check error message The exit date must be after the entry date.


39/136


Traceability of Results

Requirements

TestObjectives

Test

Cases

TestResults


40/136


Possible Project Life Cycle

TestDevelopment

Plan

Automation:- design- research

- development- testing- support

Focus on:- actions

- interface handling- technology- never on tests

Test ModuleDevelopment

Test Module

Acceptance

Plan of Approach

Execution andFollow Up

Management

Stake Holders

Business Users

IT Department

Development

TechSpecialists

intake


41/136


Examples of Time Lines

System under test becomes available in phases


Test Development

Automation

Execution and follow up

Mar Apr May Jun Jul Aug Sep Oct Nov Dec


Test Development

Automation

Execution and follow up

Sep Oct Nov Dec Jan Feb Mar Apr

Jan Feb Mar

System under test becomes end of the cycle

G


42/136


The Three Holy Grails of Test Design

The right approach per module

The proper level of test specification

The right breakdown into test modules

G il 1 B kd Di id d C


43/136


Grail 1: Breakdown, Divide and Conquer

Mission, Requirements, System Design, Use Cases,Priorities, Planning, Budget, Stakeholders, ...

Identify test modules priority and planning what, when, who, how

Development

Execution

Maintenance

Test Module 1

Analysis

Development

Execution

Maintenance

Test Module 2

Analysis

Development

Execution

Maintenance

Test Module N

Analysis

. . .

G il 1 E l b kd


44/136


Grail 1: Example breakdown

UI oriented tests does function key F4 work does listbox xyz the right values is the tab order correct

Do individual functions work like transactions in a financial system

Alternate paths in use cases

like cancel a transaction

End-to-end tests

Simulating business processes

Tests with specific automation needs like multi station tests

Tests of non-UI functions

High ambition level tests (aggressive tests)

Q ti


45/136


Question

Can you give a set ofbreakdown criteria fortests?

???


46/136

G il 1 Wh t i d b kd ?


47/136


Grail 1: What is a good breakdown?

Well differentiated and clear in scope

Balanced in size and amount

Reflects the level of tests

Modules are stand-alone, not dependent on each

other

Fitting the priorities and planning of the project

E l T t D l t Pl


48/136


Example Test Development Plan

R i P ith St k H ld


49/136


Review Process with Stake Holders

Test Team sends draft

Module to Stake Holder

Stake Holder reviews:- coverage- correctness

Stake Holder returnsnotes:- additions- corrections

Test Team receives andprocesses notes

changes needed? Stake Holder returnsnotice of approval

Test Team marks theModule as "Final"

END

no

yes

START

G il 2 A h T t M d l


50/136


Grail 2: Approach per Test Module

Organize the test design process around the test modules

Plan the test module: when to develop: is enough specification available

when to execute: make sure the functionality at action level is well-testedand working already

Process: analysis of requirements

formulation test objectives creating test cases

Identify stakeholders and their involvement: users, subject matter experts

developers auditors

Choose testing techniques if applicable: boundary analysis, decision tables, transition diagrams, soap opera testing,

...

G il 2 E th b ll S


51/136


Grail 2: Eye on the ball, Scope

Always know the scope of the test module

The scope should be unambiguous

The scope determines many things:

what the test objectives are which test cases to expect

what level of actions to use

what the checks are about and which events should generate a

warning or error (if a lower functionality is wrong)

Grail 2 E amples of Testing Techniq es


52/136


Grail 2: Examples of Testing Techniques

Equivalence class partitioning any age between 18 and 65

Boundary condition analysis try 17, 18, 19 and 64, 65, 66

Error guessing try Ccile Schfer and test the sorting of the name list

Exploratory

"Exploratory testing is simultaneous learning, test design, and test execution",James Bach, www.satisfice.com

Model-based see Harry Robinson: www.model-based-testing.org see Better Software Magazine (STQE Magazine), March 2003

Error seeding deliberately injecting faults in a test version of the system, to see if the tests catch

them handle with care, don't let the bugs get into the production version

Grail 2: More Testing Techniques


53/136


Grail 2: More Testing Techniques

State transitions

Decision tables

Soap Opera Testing

Recommended:

"Why Software Fails", James Whittaker and Alan Jorgensen, and

"Testing Computer Software", Cem Kaner, Jack Falk and Hung Nguyen

State Transition Diagram


54/136


State Transition Diagram

first check

second check

self destructdoors open

initial

signal

correct

correct

incorrect

incorrect

signaltampering

time-out

Question: what test cases would you create to cover these situations?

doors closed

alert state

State Transition Table


55/136


State Transition Table

NR STATE EVENT OUTPUT NEXT

1 doors closed signal beep first check2 doors closed tampering hi-lo alert state

3 first check correct code green doors open

4 first check incorrect code hi-lo second check

5 doors open close signal beep doors closed6 second check correct code green doors open

7 second check incorrect code hi tone self destruct

8 self destruct anything boom oblivion

9 alert state time out hi tone self destruct

10 alert state signal beep first check

Car Safety System, Model 007, Luxury Version*

*certain insurance restrictions might apply, please check with your agent

Example of a Decision Table


56/136


Example of a Decision Table

dialog data correct n y y y y

large order n n y y

online customer n y n y

show error and resume x

give discount 20% x x

give discount 3% x

notify marketing x x

send to priority execution x

display price info x x x x

TEST CASE 1

TEST CASE 2

TEST CASE 3

TEST CASE 4

TEST CASE 5


57/136

No Y2K Problems in Auckland Airport??


58/136


No Y2K Problems in Auckland Airport??


59/136


60/136

The Mechanical Approach for Test Development


61/136


The Mechanical Approach for Test Development

Start with a (preferably long) list of requirements

Make a test case for every requirement

Use a standardized test technique to translate the

requirements into test cases

Hire (many) people to perform the tests manually

.


62/136

Soap Operas


63/136


Soap Operas

Ashley hears about Jack's deposit when he thought he had

to go. Victoria lectures her father about what's wrong with him

and Nikki but Victor advises her that it's none of her businessOlivia learns Dru has no regrets about leaving and takes great

satisfaction in having Lily as her companion. Dru then asks Olivia

why she is raking Malcolm over the coals. Stopping by Gina's,Nikki spots Brad and sits with him, admitting she doesn't want to

be alone tonight. Victor stops by Mack's party at the Crimson

Lights. Ashley takes a home pregnancy test. Worried about Billy,Raul makes call and J.T. claims he doesn't know where Billy is.

Raul rushes over and finds Billy out cold in the snow. Raul worries

when he can't find a pulse. . . .

Some Other Examples


64/136


Some Other Examples

Pension Fund

World Wide Transaction System for an International Bank

William starts as a metal worker for Industrial Enthropy

Incorporated in 1955. During his career he becomes ill, works

part time, marries, divorces, marries again, gets 3 children, one

of which dies, then his wife dies and he marries again and gets 2

more children.

A fish trade company in Japan makes a payment to a vendor on

Iceland. It should have been a payment in Icelandic Kronur, but

it was done in Yen instead. The error is discovered after 9 days

and the payment is revised and corrected, however, the interest

calculation (value dating)

Properties of Soap Operas


65/136


Properties of Soap Operas

Using a recurring theme, with episodes

About real life

But condensed

And more extreme

Soap Operas for Testing


66/136


Soap Operas for Testing

Define a scope of the test to develop

Identify with the business environment

Identify which elements would make things difficult

Draft test cases (typical some dozen lines)

Write them down in modules

Soap Count the Goodies


67/136


Soap Count the Goodies

In this episode, the Goodies have won $750,000 in the

lottery. They decide to buy that big house on the corner of

the street that theyve always dreamed about. Father, BingGoody, goes to the banking office to make the necessary

arrangements. Of course, the bankruptcy that Bing had two

years ago complicates the approval, but Jim, the guy at the

bank, is very helpful and even sells Bing on the idea of a

beautiful vacation property in Mexico. Bing has a brilliantidea. Why not ask neighbor Jones if he is interested in co-

owning the vacation property? Like everything else with Mr.

Jones, his credit is perfect.

The Story Continues


68/136


The Story Continues

One week later: After a small party with some new

friends, only about $720,000 of the prize money is left,

so the mortgage arrangements need to be changed. Jim,Mr. Jones, and Bing decide to meet the next morning to

make the necessary adjustments. Since it was a great

party, they are not at their best when they meet, so many

mistakes are made. They correct some immediately,

others they miss and will need to fix later. They find thatan additional second mortgage on Mr. Joness home is

needed in order to still qualify for the vacation home.

What Will Happen Next?


69/136


What Will Happen Next?

See it all . . .

. . . in the next test module ! !

What This Soap Can Test


70/136


What This Soap Can Test

Entering a customer with a big family

Mortgage arrangements including a first and a second house

Property abroad

Both a primary residence and a vacation property

Weighing of the income of a second owner

Co-ownership on the second house only Establishing a second mortgage to finance a vacation property

Bankruptcy two years ago

Modification of the down payment

Correcting errors upon entry

Corrections after the application has been processed

Killer Soaps let me wash


71/136


Killer Soaps

Aggressive tests, aimed at finding hidden

problems

Run when everything else has passed

Ask experienced users and subject matter experts

for ideas

let me washthis thing...

Test Analysis / Test Creation


72/136


Test Analysis / Test Creation

Test Analysis

- what do we want

- what do we need

Test Creation

- confrontation with reality

- put it to the testseparate

relate

and

TESTOBJECTIVES

TESTCASES

Questions to Answer with a Test Collection


73/136


Questions to Answer with a Test Collection

mechanical soapssoaps +

objectives

1Does the system complywith the requirements *** * ***

2Are their any problems(defects and/or failures)we should know about

* *** ***

3Will the system work inpractice ** ** ***

Grail 3: Specification Level, choosing actions


74/136


Grail 3: Specification Level, choosing actions

Scope of the test determines the specification level

As high level as appropriate, as little arguments as

possible Use default values for non-relevant arguments

Clear names (usually verb + object works well)

Manage the Actions

Document the Actions

By-product of the test design


75/136

Designing Actions/Specification Level


76/136


Designing Actions/Specification Level

Scope of the test determines the specification level

As high level as appropriate, as little arguments as

possible Use default values for non-relevant arguments

Clear names (usually verb + object works well)

Manage the Actions

register them, keep them organized

Document the Actions

intention, arguments, assumptions, conditions, implementation notes, ...

Levels of Actions


77/136


Levels of Actions

High level actions to capture functions in abusiness process hide unneeded details

always hunt down details usually implemented with lower level actions

Low level actions to access the system interface either built-in in a action based tool like TestArchitect

or tailor created as part of an interpretation harness, usingscripting and/or record&playback

Example of Going High Level


78/136


a p e o Go g g e e



79/136


p g g

section Get cost valueskey

key navigate F7

key navigate 5

window

wait for controls loaded view account infotext

check breadcrumb # view account info breadcrumb

window field

check focus view account info cancel

start cost cost in cost out net cost

capture cost values >> starting cost >> total cost in >> total cost out >> total net cost



80/136


p g g



81/136


p g g

section Capture initial number

key

key navigate F7

key navigate 3

page tab

locate page tab Scan Criteria

window

wait for controls loaded search

text

check breadcrumb # search breadcrumb


select search scan direction Backward


enter value search business date match # bus date

source control

click search go

window

wait for controls loaded search results

store ascapture sequence num >> seq num



82/136


p g g


83/136


Test Automation

TESTAUTOMATION

TEST

DEVELOPMENT


Components of Automation


84/136


p

Tests and test data what needs to happen to prove or disprove the functionality under

test best to keep this separate from the technical part of the automation

Programming/scripting technical specification of the individual steps of the automation

Interface information what is where on screens, web pages, etc shield volatile physical details

UI and non-UI automation functions the actual (usually tool provided) commands to work with the

interfaces of the system under test

Automation: Implementing Actions


85/136


p g

In a keyword driven approach the automationfocuses on the actions: as opposed to programming test cases

in general this will lead to fewer and shorter scripts etc in theautomation

In the Action Based Testing method actions are

implemented in three ways: built-in in the tool, like click, enter, check etc

defined in an action definition, using other (lower level) actions similar to functions in a programming language

programmed in a programming or scripting language


86/136

Automation Overview


87/136


Actions

Test Modules

ActionDefinitions

Low Level

Actions

InterfaceDefinitions

(UI, SQL, OLE, ...)

Test Architect

Built-in

Additional

Scripting (C#, VB, ...)

90% +

split up in lower level actions provided defaults values for

arguments

shield (volatile) technicaldetails of an interface

provide "logical" names forcontrols and their classes

test objectives test cases

Doing the Count Down . . .


88/136


g

6000

250

Test Cases Actions ProgrammedActions

3000

22

200

20

In a good application of the keywords approach a large increase in test cases (like doubling the amount)should result in a modest increase in actions, and a minor increase, if any, in programmed actions.

Programming


89/136


g g

Often called "scripting", but much like regular programming

Traditional playback tools mostly have built-in scripting

languages often Basic look-a-likes, WinRunner's TSL is a C look-a-like

More and more tools support existing programminglanguages like Visual Basic

Regular languages can in some cases do the job too in particular for non-gui's for example scripting languages like Python and Ruby but also Java, C++, VB, C#, etc

Functionality of an Action Interpreter


90/136


Function Interpret

While not end of file doRead next line

Split the line into arguments

Look up the action in the "action list"

Execute the action

Print the results of this line in the reportEnd of the while loop

End

ReportInterpreter

Atomic and Generic


91/136


In ABT key recommendations for programmedactions are to make them: atomic, only one small task per action

generic, maximum re-use

Avoid programming test cases or high level

actions split them up in well-defined low level actions

Automation: example test lines


92/136


System Under Test Test Lines

Automation, step 1: make "Action Functions"


93/136


identify theUI element

perform theoperation

identify theUI element

perform thecheck

Automation, step 2: "Divert" to the action functions


94/136


call this function for each line in the test module

Interface Information and Interface Handling


95/136


Handling interfaces is the essential challenge automation works "outside in"

interfaces are not necessarily designed with test tools in mind

Common interfaces GUI

Web

API's, with methods and/or functions message protocols, on basis of TCP/IP or SS7

embedded software

databases with SQL

files (like in batch runs)

Identifying Elements of an Interface


96/136


All elements have properties can be used for identification and verification

Some elements can be easily identified like buttons, checkboxes, frames

Other elements have only volatile and/orambiguous properties like text boxes, list boxes, combo boxes

can differ across runs, or across versions/builds

Text labels tend to be "invisible" only bitmap

Interface Definitions


97/136


Quickly define how interfaces must be accessed

Commonly used for GUI's, however applicable toother interfaces as well

Typically kept in a table or spreadsheet

separate from the test modules

Components of Interface Definitions


98/136


Interface entity window

web page

text screen 3270 page (typically based on the HLLAPI standard)

class member

api function

message database table

Interface element

button text box

link

field

argument ....

Example Interface Definition


99/136


window control valueenter add dialog number 1234567

enter add dialog name saw

enter add dialog description Cut through things

enter add dialog quantity 500

source controlclick add dialog ok

INTERFACE ENTITY add dialog

interface entity setting title Add A New Item

ta name ta c ass a e

interface element number textbox Item Number

interface element name textbox Name

interface element description textbox Descriptioninterface element quantity textbox Quantity On Hand

ta name ta class caption

interface element ok button Process

interface element cancel button Cancel

Some Tips to Get Stable Automation


100/136


Make the system under test automation-friendly developers are not always inclined to do that, but it pays off ask to add specific property values to the GUI interface controls for automated

identification like accessible name in .Net and Java, or id in Web controls

Use "active" timing wait based on conditions in the system under test, avoid fixed sleeps

Test your automation develop separate tests for the actions themselves

Use automation to identify differences between versions of the systemunder test

test in a loop if all control can still be found

Keep an eye on the test design unsuccessful automation virtually always can be traced back to poor test design

"Accessible Name" property


101/136


Testing the automation . . .


102/136


The automation is a service provider to the testers: provide working actions

provide accurate up to date interface definitions

In each project consider having the following

folders:

Automation Tests

Action Tests

Interface Tests

Cross Platform Testing


103/136


No longer absolute dominance of Windows and IE

Diversity in: operating systems

Windows (XP, Vista, Windows 7), Linux (variety of distributions), Mac widget toolkits

.Net, InfraGistics, QT, Java, STK+, Flex, ...

browsers Internet Explorer, FireFox, Safari, Chrome,... within browsers: Ajax, JavaScript, Java,

VB Script, Flash, ...

non-traditional platforms Windows Mobile, Android, Symbian, RIM, ... Netbooks!

non-client platforms

Cloud computing, SOA, ...

This trend will continue: even if Windows and/or IE keep dominating the alternatives can't be ignored much

longer, diversity is the name of the game

Cross Platform Automation


104/136


The key ingredient for success is design test design is even more the decisive factor

automation needs to be layered and flexible

tooling needs to plug in into the overall model

UI LEVEL TESTS

HIGH LEVEL ACTIONS

LOW LEVEL ACTIONS

GENERIC CONTROLS AND OPERATIONS

PLATFORM 1 PLATFORM n. . .

HIGH LEVEL TESTS API LEVEL TESTS

Non-UI Testing


105/136


Examples application programming interfaces (APIs)

embedded software

protocols files, batches

databases

command line interfaces (CLIs)

multi-media

Impact is mainly on the automation test design should in most cases be transparent towards the

specific interfaces

Multiple System Access


106/136


System (part)Under Test

Automation Scheme

APIaccess

protocolaccess

UIaccess

databaseaccess

Test Modules, driving eitherone or multiple interfaces

Device Testing


107/136


Software Under Test

MiniAgent

ABTAutomation

InterfaceInfo

Testing HostDevice

Testing a Service Architecture or Protocol


108/136


Automation

InterfaceDefinitions

stash(optional)

CLOUD,SOA, ...

SEND

RECEIVE

Testing the Cloud, testing the SOA


109/136


A lot of business as usual for testing

Test everything always puts an extra need on high coverage automation

not necessarily time to "maintain" not-working automation

Where is what, and is it safe is data where you think it is, and what you think it is are protocols and safety measures working

Various perspectives unit level, service level, client level, user level testing functions, testing interfaces, testing protocols

Expect the unexpected, "jungle testing"

We still have to learn

"Jungle Testing"


110/136


Expect the unexpected prone to unexpected requests prone to unexpected situations in data prone to attacks

how does a generic design respond to a specific unexpected event?

Difference in thinking coding bug: implementation is different from what was intended/specified jungle bug: system does not respond well to an unexpected situation

To prepare study the matter (common hack attacks, ...) make a risk analysis

make time to discuss it (analysis, brainstorm) involve people who can know use "exploratory testing" (see James Bach's work on this) use an agile approach for test development applies equally to testing, requirements, design

testing should focus on the specific attacks

From the Headlines


111/136


Bugs can be data driven

They can happen in real time the earlier you see them the better

There will be material and immaterial damage Your tests won't be perfect

sometimes unexpected simply means unexpected

learn, generalize, prevent the bad bug, and its family, for the future 20/20 hindsight is an acceptable technique...

Handling Batch Processes


112/136


Transfer Jones Williams 100Check Balance Jones -100Check Balance Williams 100 Jones Williams 100

Batch Run

Jones -100

Williams 100

Test Module

"Delayed Checks"

Batch Input File

Batch Output File

Check Balance Jones -100Check Balance Williams 100

1

1

2

2

3

Multi-media: The "Play List" Approach


113/136


The testers focus on a play list items are played/shown while the machine tells what it is the tester

should see/hear

the tester acknowledges/falsifies the system stores checksums of those items that were ok

efficient and relatively easy to store and handle

Checksums can be kept for files, bitmaps etc

Next time only the changed items go on the playlist again when the checksum is different from the previous run/version

in most systems this will only be a limited amount of items

Performance Testing


114/136


The topic is complex, many components andphenomena

To formulate tests in actions can be surprisinglystraightforward

Often performance testing isn't testing, but moreclose to research

The three controls you can/should address:hardware (equipment, infrastructure, etc)software (programs, database models, settings, etc)demands (1 second can be 10 times more than 2 seconds)

See also: "Web Load Test Planning", Alberto Savoia, StickyMinds.com

Data Driven Testing


115/136


Separate test logic from the data

Have the data, including expected outcomes, in a

separate file or table can also be generated during test execution, for example extracted

from a database

Use placeholders in the test case

Works for both scripted and keyword drivenapproaches

Data Driven Testing, Example


116/136



117/136


Topics Test Management

TESTAUTOMATION

TESTDEVELOPMEN

T


Organization


118/136


Much of the success is also gained or lost in how you organize theprocess who does test design who does automation

what to outsource, what to keep in-house

Write a plan of approach for the test development and automation scope, assumptions, risks, planning methods, best practices

tools, technologies, architecture stake holders, including roles and processes for input and approvals team . . .

Assemble the right team testers, lead testers automation engineer(s) managers, ambassadors, ...

Test design is a skill . . .Automation is a skill . . .

Management is a skill . . .


119/136

Typical Time Allocation


120/136


TEST DEVELOPMENT

AUTOMATION

time

efforts

Embedding QA


121/136


Clear division of tasks and responsibilities: testers (who will now become test developers)

users, power users, subject-matter specialists

test automation engineers (do they work for you?) developers

other stakeholders

Decision making and communication who's call is it to pass a system for release what happens with bugs

who does which testing

what whistles to blow/not blow, when

Flexibility testing is a "2nd order dependency" activity

firmness on the integrity of provided information (but be practical)

Question


122/136


What are good skills for atester?

???

Skills for the Team


123/136


Typical skills needed testing (real testers)

subject matter

beware of "guru's", don't allow singleindividuals to monopolize key knowledge

technology (often underestimated and then weak spot)

engineering practices (like configuration management)

soft skills (communication, managerial, diplomacy)

Acquiring the skills

training, reading, practicing, coaching interest, curiosity

hiring, borrowing

overall picture counts, not everyone needs to know everything

The Tester as a Detective


124/136


Looking for the bad guys (bugs)

Specific personality traits: authority, respected

inconspicuous, not threatening

able to look past excuses and stories

mindful of details

Mr. Authority

Mr. Inconspicuous Mr. No Nonsense Mr. Details

Tester Needs Hunting Instincts


125/136


Off-shoring


126/136


Can a keyword drivenapproach be done off-

shore? If yes, what would it take?

If no, why not?

???

Off-shoring Keyword Automation


127/136


Potential can get a lot more tests at a lower cost

potential of double cost benefit: 1. off-shore, 2. automation risk of double backlash: 1. bad design, 2. bad automation

flexibility, particularly for "bursty" projects ramp up quickly, scale back quickly

Pitfalls little "ownership" with the system under test and the subject matter under

test a keyword method is a subtle approach (see the rest of this tutorial...). Thismakes outsourcing challenging: mind the "Automation Cliff" . . .

very hard to go back when things turn out disappointing

What to do keep local ownership provide clear direction, constant attention and coaching supervise, supervise, supervise ask to create example test modules first and review them carefully

Organization and Project Level


128/136


ACTIVITIES AT PROJECT LEVEL

ACTIVITIES AT ORGANISATION LEVEL



Beyond the projects: Test Governance


129/136


TEST COMPETENCE MANAGEMENT(improvement focus)

initiation, know how, support, ...

experiences, products, people, ...

pilot projects

training

coaching

resource mediation

test quality assurance

test environments

testware management

vendor contacts

TEST BUSINESS POLICIES AND DIRECTIONS(significance, position, organization)

managem

ent&contr

ol

TestMaintenance

TestExecution

TestDevelopment

TestDesign

Test Strategy& Planning

TEST PROJECT MANAGEMENT(solution focus)

Test Policy


130/136


Corporate point of view on testing (or software quality)

Not a common topic in many organisations

However can be useful to have

Policy Model


131/136


Business Environment

Organisation Environment(Including Policies)

IT OrganisationAnd Policies

Organisation andPolicies of Testing

How to Get It


132/136


Think about it yourself

Identify the stake holders

Use existing policy documents (if any)

Discuss it: Within the testing department Externally (IT, business, stakeholders)

If possible start low profile and end high profile

Policies should be recognized and understood

Consider the use of an incremental document

Typical Issues (1)


133/136


What is the significance of testing and what may itcost

Connection to critical success factors of thecompany

Do we have problems

When should testing be done in the life cycle

Who should be involved (test development,assessment, reporting)

Who is responsible

Typical Issues (2)


134/136


How to deal with testing expertise

Centralized / decentralized

Methods and tools

Migration to a desired situation

Automation or not

Examples of policy statements


135/136


Testing and test automation are regarded as businesscritical

Testware (developed tests and automation) is to be treatedas business asset

User friendliness is a major priority in testing

Test Automation is used as much as possible

QA is responsible for timely and accurate information on

system quality, independent of other stakeholders

System Development is responsible for system quality

Releasing a system is a business responsibility


136/136

Documents

Delivering Test Automation Success