Does my test script smell?

Detect, fix and preventcommon maintenance issues

with automated tests

Martin GijsenTest automation architect

TESTWAREZ 2009© 2009

Overview● Maintenance of automated tests● Test script smell● Common maintenance issues● Conclusion● Q & A

Automated testing in practice

About 2/3 of all test automation projects fail sooner or later– Brian LeSuer(testing experience magazine,December 2008, page 47)

2

1

time

total cost / effort

1. manual testing2. effective automated testing3. less effective automated testing

3

Maintenance of automated tests● The Record & Playback approach does not work● Total maintenance depends on:

● Number of changes● Maintenance sensitivity (does change affect test?)● Maintainability (how much effort)

● For continuity of benefits, focus on:● 3 R's: Tests must be easy to write, review and revise● Maintainability and maintenance sensitivity

Test script smell (1)● Test script smell is derived from 'code smell'● Wikipedia:

● In computer programming, code smell is any symptom in the source code of a program that possibly indicates a deeper problem.

● Code refactoring is the process of changing a computer program's internal structure without modifying its external functional behavior or existing functionality.

Test script smell (2)● Test script smell is any symptom in an

automated test that possibly indicates a deeper problem.

● Test refactoring is the process of changing the internal structure of an automated test without modifying its external functionality.

● Most test script issues increase maintenance● Focus on the average tester!

Common maintenance issues● Unreadable tests● Long test cases● Repeating sequences● Interface details in test cases● Tooling details in test cases● Literal constants in test cases

Examples will use the freeware ETA Framework

A Selenium test (Java)Selenium selenium = new DefaultSelenium (“localhost”, 4444, “*firefox”, “http://www.google.com”);

selenium.start ();

selenium.open (“/”);

if (selenium.getTitle ().indexOf (“Google”) < 0) {

reportError (“an error message”);

}

selenium.type (“q”, “the quick brown fox jumps over the lazy dog”);

selenium.click (“btnG”);

if (selenium.getTitle ().indexOf (“the quick brown fox jumps over the lazy dog”) < 0) {

reportError (“an error message”);

}

selenium.click (“link=YouTube”);

A keyword based testnew DefaultSelenium localhost 4444 *firefox http://www.google.com

start

open /

getTitle ().indexOf (“Google”)

type q the quick brown fox jumps over the lazy dog

click btnG

getTitle ().indexOf (“the quick brown fox jumps over the lazy dog”)

click link=YouTube

!

Interface details removednew DefaultSelenium localhost 4444 *firefox http://www.google.com

start

open /

getTitle ().indexOf (“Google”)

type ?queryField the quick brown fox jumps over the lazy dog

click ?searchButton

getTitle ().indexOf (“the quick brown fox jumps over the lazy dog”)

click link=YouTube

!

Tooling details removedtype URL

open browser firefox www.google.com

open /

check page title Google

type ?queryField the quick brown fox jumps over the lazy dog

click ?searchButton

check page title the quick brown fox jumps over the lazy dog

select link YouTube

!

!

!

Abstraction level increasedtype URL

open browser firefox www.google.com

web page titleopen page / Google

search stringsearch the quick brown fox jumps over the lazy dog

link textselect result YouTube

The automated test

Test caseTest case

Scripted proceduresScripted procedures

Source codeSource code

Test analyst

Test analyst / Developer

Developer

ApplicationApplication

Conclusion● Abstraction is the key to low maintenance● For easy maintenance:

● Make tests readable (to your manager?)● Make tests compact (high abstraction level)● Remove interface + tooling details (implicit or config)● Use symbolic constants

● Define test instructions top down● Test automation is software engineering● Example test engines: ETA Framework & FIT

Q & A

?

Main sheet & configuration sheetname

run sheet configurationrun sheet procedures

type URLopen browser firefox www.google.com

namerun sheet test cases

Main sheet:

Config sheet: name valuedefine constant queryField qdefine constant searchButton btnG

Proceduresname Param 1 Param 2

define procedure open page url pageTitle

url

open URL ?url

title

check page title ?pageTitle

end procedure

Proceduresname Param 1

define procedure search searchString

input field value

type ?queryField ?searchString

item name

click ?searchButton

title

check page title ?searchString

end procedure

Configuration sheet

The test (case)identification description

begin test case google1 A Google test

web page titleopen page / Google

search stringsearch the quick brown fox jumps over the lazy dog

link textselect result YouTube

end test case

close browser