BIG Testing' with Hans Buwalda

www.eurostarconferences.com

Today’s webinar is presented by Hans Buwalda and he will discuss BIG Testing.

Welcome to the EuroSTAR July Webinar Series

This webinar is due to start at 4pm. Make sure you stick around at the end for the Q&A session and continue the conversation with the speaker on Twitter after the show!

@esconfs#esconfs


BIG Testing

Hans Buwalda, CTO, LogiGear (USA)An internationally recognized expert in testing, Hans Buwalda is the pioneer of keyword-driven test automation, an approach now widely used throughout the

testing industry. Working in The Netherlands for CMG (now CFI) he helped develop the successful TestFrameTMframework. He now is CTO of LogiGear Corporation in California, responsible for the keyword-driven Action Based

TestingTM method, and the tools that support it: TestArchitectTM and TestArchitect for Visual Studio®. Hans speaks frequently at international conferences and is

coauthor of Integrated Test Design and Automation: Using the TestFrame Method.

@esconfs#esconfs

Q&A


If you have a question, the control panel on the right has a Questions box. All questions will be answered at the end of the presentation.

The GoToWebinar attendee interface is made up of two parts. The Viewer Window shows the presenter’s screen. The Control Panel is where attendees can interact and ask questions.

The webinar will be recorded and you will be notified via email when it is available.

@esconfs#esconfs


How can I get the slides?

Slides and all materials will be posted on the EuroSTAR website as well as emailed to you.

@esconfs#esconfs


Join the conversation on Twitter

#esconfs@esconfs

@hansbuwalda

Continue the conversation with the speaker on Twitter after the show!

@esconfs#esconfs

BIG Testing

Mr. Playback

Hans BuwaldaLogiGear

Tutorial excerpts

Keywords, Test Design, Automation,

Virtualization, Globalization

© 2013 LogiGear Corporation. All Rights Reserved

About LogiGear

Software testing company, around since 1994

Testing and test automation expertise, services and tooling- consultancy, training- test development and automation services- "test integrated" development services

Aims to be thought leader, in particular for large and complex test projects

Products:- TestArchitect™, TestArchitect for Visual Studio™- integrating test development with test management and automation- based on modularized keyword-driven testing

www.logigear.comwww.testarchitect.com


What is "BIG"

Big efforts in development, automation, execution and/or follow up

It takes a long time and/or large capacity to run tests (lot of tests, lot of versions, lot of configurations, ...)

Scalability, short term and long term

Complexity, functional, technical

Number and diversity of players and stakeholders- pigs, chicken, elephants, ankle biters, ...

Various definitions of "big" possible... and relevant...- "10 machines" or "10 acres"- "1000 tests" or "1000 weeks of testing"

Big today means: big for you- "non trivial", you need to think about it

"Windows 8 has undergone more than 1,240,000,000 hours of testing"

Steven Sinofsky, Microsoft, 2012


Keywords, essential for scalability

Distinguish tasks for test development and for automation

The test developer creates tests using "actions". Each action consists of a keyword ("action word") and arguments

Automation focuses on automating the actions, each action is automated only once

however…. actions by themselves are just a format, not a method….


Action Based Testing

Test Development Plan

Test Cases

TestObjectives

Test Module 1

Test Cases

TestObjectives

Test Module 2

Test Cases

TestObjectives

Test Module N

Actions

. . .

ACTION AUTOMATION

Break down

Automate


Test Design

Effective test breakdown (into test modules)- make sure test modules (test clusters) have a clear focus- keep different kinds and levels of tests separate

Right level of actions- as “high level” if possible, hiding as many details as much as possible- ...but not if the details are relevant for the test

It is my believe that successful automation is not a technical challenge. It is most of all a test design challenge.


What's the trick...


What's the trick...

Have or acquire facilities to store and organize you content

Select your stuff

Decide where to put what- assign and label the shelves

Put it there

If the organization is not sufficient anymore, add to it or change it


Example of a business test module

Test a business functionality, doesn't show details that are not relevant to that Navigation details are avoided, unless they're meant to be tested

TEST MODULE Car Rental Payments

user

start system john

TEST CASE TC 01 Rent some cars

first name last name car

rent car John Doe Ford Escaperent car John Doe Chevvy Volt

last name amount

check payment Doe 140.4

FINAL

close application


Example of an interaction test module

In this "low level" test details are not hidden, since they are the target of the test The right level of abstraction depends on the scope of the test, and is an outcome of your test

design process

TEST MODULE Screen Flow

user

start system john

TEST CASE TC 01 "New Order" button

first name control

click main new order

window

check window exists new order

FINAL

close application


Eye on the ball, Scope

Always know the scope of the test module

The scope should be unambiguous

The scope determines many things:- what the test objectives are- which test cases to expect- what level of actions to use- what the checks are about and which events should generate a warning or

error (if a “lower” functionality is wrong)


What I have seen not work

"Over-Checking": having checks that don't fit the scope of the test

Forcing data driven: making all tests data driven (variables, data files) without clear reason

Combinatorial explosions: test all ... for all ... in all ...

All actions high level (or all actions low level)

Many tests for forms and dialogs, little tests for business processes

Abundance of irrelevant comments, and lack of relevant comments


Choosing actions

Scope of the test determines the actions

As high level as appropriate, but not too high

As little arguments as possible- use default values for arguments not relevant to the scope of the test

Clear names (usually verb + noun usually works well)- to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify

client" (or vice versa)- tests are not C++ code: avoid "technical habits", like mixed case and underlines

Manage the Actions

Document the Actions

Actions should be by-product of the test design


Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test

key

key navigate F7key navigate 3

page tab

locate page tab Scan Criteria

w indow

wait for controls loaded search

text

check breadcrumb general functions > search

w indow control value

select search scan direction Backward

w indow control value

enter value search business date match # bus date

source control

click search go

w indow

wait for controls loaded search results

w indow control variable

get search results sequence number >> seq num


variable

get sequence number >> seq num

Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test


Low-level, high-level, mid-level actions

Low-level: detailed interaction with the UI (or API)- generic, do not show any functional or business logic- examples: "click", "expand tree node", "select menu"

High-level: represent a business function specific to the scope of the test- hide the interaction- examples: "enter customer", "rent car", "check balance"

Mid-level: auxiliary actions that represent common sequences of low level actions- usually to wrap a form or dialog- greatly enhance maintainability- example: "enter address fields"

enter customer

enter address fields

enter select set . . .. . .


Test Development and Automation in sprints

Test ModuleDefinition(optional)

Test Module Development

Interface Definition

Action Automation

Test Execution

Sprint ProductsProduct Backlog

Test re-use

Automation re-use

product owner

teamprod owner &

team

User storiesDocumentation

Domain understanding

Acceptance CriteriaPO Questions

SituationsRelations

Agile life cycle

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules


Some Tips to Get Stable Automation

Make the system under test automation-friendly

Use "active" timing- always wait for something, never just wait a fixed amount of time

("passive timing")

Test your automation

Use automation to identify differences between versions of the system under test

Keep an eye on the test design


Look for properties a human user can't see, but a test tool can This approach is a must-do for speedier and more stable automation

- interface mapping is often bottleneck, and source of maintenance problems- with predefined identifying property values an interface map can be created without "spy" tools- not sensitive to changes in the system under test- not sensitive to languages and localizations

Examples:- "id" attribute for HTML elements- "name" field for Java controls- "AccessibleName" property in .Net controls (see below)

Automation-friendly design: hidden properties


Environments, configurations

Many factors can influence details of automation- language, localization- hardware- version of the system under test- system components, like OS or browser

Test design can reflect these- certain test modules are more general- others are specific, for example for a language

But for tests that do not care about the differences, the automation just needs to "deal" with them- shield them from the tests


Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there).Can be a feature in a test playback tool, or something you do with a global variable or setting.

Variation Variation Variation

"Variations"

"Master Switch"

Actions, Interface Definitions

. . .


Possible set up of variations

linked variation

keyworded variation

Specify for example in a dialog when you start an execution:


Virtualization

Virtual machines rather than physical machines- allow "guest" systems to operate on a "host" system- host can be Windows, Linux, etc, but also a specialized "hypervisor"- the hypervisor can be "hosted" or "bare metal"

Main providers:- VMWare: ESX and ESXi- Microsoft: Hyper-V (readily available in Windows 8)- Oracle/Sun: Virtual Box- Citrix: Xen (open source)

Hardware support gets common now- processor, chipset, i/o- for example "slat" (second level address translation, helps map addresses to memory)

For most testing purposes you need virtual clients, not virtual servers- most offerings in the market currently target virtual servers, particularly data centers

Virtual clients will become more mainstream with the coming of VM's as part of regular operating systems- Windows 8: Hyper-V- Linux: KVM


Virtualization, a testers dream...

In particular for functional testing

Much easier to define and create needed configurations- you basically just need storage- managing this is your next challenge

One stored configuration can be re-used over and over again

The VM can always start "fresh", in particular with - fresh base data (either server or client)- specified state, for example to repeat a particular problematic automation situation

Can take "snap shots" of situations, for analysis of problems

Can use automation itself to select and start/stop suitable VM's- for example using actions for this- or letting an overnight or continuous build take care of this


Virtualization, bad dream?

Performance, response times, capacities

Virtual machine latency can add timing problems- see next slide- can be derailing in big test runs

Management of images- images can be (very) large, and difficult to store and move around

• there can be many, with numbers growing combinatorial style• configuration in the VM can have an impact, like fixed/growing virtual disks

- distinguish between managed configurations and sandboxes- define ownership, organize it- IT may be the one giving out (running) VM's, restricting your flexibility

Managing running tests in virtual machines can take additional efforts on top of managing the VM's themselves- with the luxury of having VM's the number of executing machines can increase rapidly- one approach: let longer running tests report their progress to a central monitoring service (various

tools have features for this)


Virtual machines, capacity

Key to pricing is number of VM's that can run in parallel on a physical machine

An automated test execution will typically keep a VM more busy than human use

Factors in determining VM/PM ratio:- memory, for guest OS, AUT, test tooling- storage devices (physical devices, not disk images)- processors, processor cores- specific hardware support (becoming more common)

• processor, chipset, I/O

We started regression with 140 VMs.Very slow performance of

Citrix VM clients.


Globalization

Three Challenges:- other countries, other cultures- geographic distances- time differences

Seven "Patterns":- "Solution"- "Push Back"- "Time Pressure"- "Surprises"- "Ownership"- "Mythical Man Month"- "Cooperation"

other country

surprise


Three Challenges

Other country- tricky, but can be managed- study and prepare (culture, governance, etc)- have a clear company culture to complement regional

(and professional) culture

Distance- cumbersome- have good (great) communication features, at all levels

Time difference- this is the biggest problem, comes back every day- as much as possible have local management and responsibility

company culture


Seven patterns

"Mythical Man Month"- more (inexpensive) resources does not mean faster timelines

"Solution"- team "solves" problems rather than asking

"Push Back"- on-shore side dictates work and practices in detail- tell teams to push back if necessary

"Time Pressure"- deadlines are quite often imposed without need

"Surprises"- bad news is communicated too late

"Ownership"- establish clear ownership: shared responsibility is no responsibility

"Cooperation"- distance creates confusion and conflict (human nature)- tip: work with briefs and agendas for meetings


Industrial Organization

Large scale testing can move from a "design" to a "production" focus- mostly applies to test execution, but also seen for test development- this not black and white, both paradigms can occur in the same projects

A production organization is different a development organization- different professional culture- emphasis more on delivery and scale, "thinking big"- discipline rather than creativity, "get stuff done"- activities are like planning, control, logistics, information

Responsibilities- Keeping the tests running- Allocating resources- Respond to hick-ups- Analyze and address automation issues


Summary

Not all "big project" challenges are the same

Think before you do. Best results come from combining effective concepts, tricks and tools

Consider tests and automation as products that need planning and design

Virtualization can be a solution, but also a challenge

Globalization can help scalability, but needs knowledge and attention

Big testing projects typically have a "production" focus, in addition to test development and automation


Questions?

@esconfs#esconfs

EuroSTAR Webinar Archive

Access almost 70 on-demand software testing webinars by the world's leading testing experts. Topics include….


Agile, Automation, Test Management, Process Improvements, Risk-based Testing, Performance Testing, Test Documentation, Cloud Testing, Session-based Testing, Weekend Testers

Visit the archive!

http://www.eurostarconferences.com/community/member/webinar-archive

@esconfs#esconfs

The EuroSTAR Community.Are you a member?

Did you know we have an online resource area with almost …


600 software testing presentations, 70 topical webinars, over 40 videos and podcasts and 25 expert eBooks

Do you know that membership is completely free and you can access these premium testing resources anytime?

Join today! http://www.eurostarconferences.com/

community/member

@esconfs#esconfs


Are you coming to Gothenburg?

Join us in the Swedish Exhibition & Congress Centre, Gothenburg,

Sweden from the 04-07 November for the 21st annual EuroSTAR

Conference.

Visit www.eurostarconferences.com for more information on what you

can expect to see at Europe’s largest software testing event!

@esconfs#esconfs

Book before Sept

http://www.eurostarconferences.com/

Have you seen the latest competition?

You and 3 of your team could be joining us at the 2013 EuroSTAR Conference!Entering is easy! Just send us a short video (2 minutes approx.) demonstrating how your team works together to meet

goals or solve problems.

Deadline for entries is Wednesday, August 14th! More information here: http://www.eurostarconferences.com/content/teamstar-2013@esconfs

#esconfs

http://www.eurostarconferences.com/content/teamstar-2013

Don’t miss these great savings, book your ticket today!

Plus Make even bigger savings when you register for the Group discount where every 5th attendee goes free!

For more information on the Conference Programme and pricing options go to www.eurostarconferences.com@esconfs

#esconfs

Technology

BIG Testing' with Hans Buwalda