Making Visual GUI Testing Agile

2016-04-28Geoff Bache

Today’s take home points

• Arrange, Act, Assert: separate problems, different techniques/tools!

• Capture system behaviour, filter it, and manage changes in ito This is called ”Approval Testing” and comes with tool support!

• Have multiple independent ways of locating GUI widgets available. o This is called a ”Multilocator”

2

Text Image Position

Visual GUI Testing Fact Sheet

• How it workso Treats the GUI like a bitmapo Use the mouse and keyboard to control ito Makes use of image recognition and text recognition (OCR) techniques

• Examples:o Open source: Sikulio Commercial tools: JAutomate, Eggplant, UFT etc

3

Why Visual GUI Testing?

• Advantageso Independent of GUI technologyo Can simulate a user very closelyo Easy to get going quickly

• Disadvantageso Fairly immature and ”bleeding edge”o Brittle and un-Agile?

4

A favourite Sikuli example: automation or testing?

5

Sikuli’s advice on how to make a test

6

Unit Test Case: Arrange-Act-Assert

7

Arrange Act Assert

x = Number(1);y = Number(2);

x.add(y); assert x.value() == 3;

Visual GUI Test Case: Arrange-Act-Assert

8

Arrange Act Assert?

Assertions in GUI tests

One assertion per test, now!

assert answerToLifeTheUniverseAndEverything() == 42;

Or, the scattergun approach:

assert textInCell(1, 1) == ”Geoff”;

assert textInCell(3, 4) == ”Sweden”;

assert exists(”OurLogo.png”);

assert not exists(”DancingPony.png”);

...

9

Approval Test Case: Assert = Collate-Filter-Compare

10

Arrange Act

Assert

dump

copy

pdf2text

xmllint

CompareFilterCollate

Behaviour change!

11

Approved Output New Output

TEST FAILURE

Scenario 1: It’s a bug, go fix it!

12

Approved Output Buggy Output

TEST FAILURE

Scenario 2: It’s a feature, press ”Approve”!

13

Old Approved Output New Approved Output

TEST FAILURE

Scenario 2: it’s a feature

14

Old Approved Output New Approved Output

overwrite

Filtering the text

15

\d\d\d\d-\d\d-\d\d{REPLACE <date>}

Tool support for Approval Testing

● Open Source● Approval Testing● Data-Driven Testing● Test Management● Mature (since 2004) and feature-rich● I am the author

http://texttest.orgOther tools:

• Approval Tests (Llewellyn Falco): http://approvaltests.sourceforge.net/

• Pearlfish (Nat Pryce): http://github.com/pearlfish/pearlfish-java

16

Approval Testing with Sikuli

17

Arrange

Act Assert

dump

copy

pdf2text

xmllint

CompareFilterCollate

Text Image Position

”Sikuli is an Automation tool, not a test tool”• Don’t compare screenshots

o What should the exactness be?- High -> brittle- Low -> green tests when it’s broken

• Use Sikuli for the ”Act” part onlyo Can use low exactness without fear

• Don’t use the Sikuli IDEo Normal Python development with normal toolso Shared image repository

18

Reliability and maintenance issues

• find( )o Image recognition using OpenCVo change screen, font, theme? Underscore, shading?o capturing, administering a lot of images

• find(”Ny beställning”)o Text recognition using Tesseract (”experimental”)o White text on dark background, italics, spaces etco Finds similar text elsewhere in the GUI

• find(13, 512)o Pixel position on screen!o Unreliable. Since 1980s.

19

Multilocators• Presented by Filippo Ricca (University of Genoa) at ICST 2015

o Web testing using Selenium. Tests were unstable.o Selenium uses XPaths for finding elements in a web GUI o 5 algorithms to make an XPath from an element. All may break when GUI

changes.o Use all 5. ”Vote” on where the element can now be found.o Update the ones that failed based on the ones that succeeded.o Automatic test maintenance! Not foolproof, but saves time.

20

Multilocators for Visual GUI Testing• We have 3 visual methods for finding GUI elements

- Text recognition (Tesseract), image recognition (OpenCV) and the pixel position

• All of these are unreliable and hard to maintain if used alone

• If we use them all, they can compensate for each othero We can update screenshots based on text recognition o We can do a textual scan of a screenshot to improve our text search

• They can speed each other upo The position can always be recorded and serves as a hint where to search first

21

Text Image Position

Combined method for finding GUI-objects• Always start looking for text, if possible

o clickText(”Ny beställning”)

• If this works, build a multilocator automaticallyo Generate a screenshot ->o Store the location in the window -> xhint=2-94 of 1920, yhint=509-528 of 1168

• If it doesn’t work, ask the user to take a screenshot by hand

• Next time, search for the screenshot in the area stored from last timeo Very fast if it workso Search the whole window again otherwise

• If it fails, we can alsoo Try again to find the text, first in the hint area and then the whole windowo Click in the hint area anyway, store the new text and screenshot there, and if the test works,

update everything automatically

• Manual maintenance when objects change appearance and move simultaneously

22

In real life: Evry’s medical referral system • 120 tests that run every night

o ”Arrange”: Carefully created minimal databases, stored as text files, scriptingo ”Act”: Hierarchically defined ”usecases” in Python, using Sikuli and

Multilocatorso ”Assert” : Approval testing with TextTest

- Database dumps, asciified HTML views, internal logs and XML sent to other systems

o Random failure rate about one every other run (~0,5%)- Timing- Windows 8’s Metro interface...

23

Text Image Position

How to make Visual GUI Testing Agile

• Arrange, Act, Assert: separate problems, different techniques/tools!

• Capture system behaviour, filter it, and manage changes in ito This is called ”Approval Testing” and comes with tool support!

• Have multiple independent ways of locating GUI widgets available. o This is called a ”Multilocator”

24

Text Image Position

25

26