Agile automation-practices by Shadi Mousa Galil Software

1 Confidential and propriety Galil Software, Ltd. 2014 1-Jun-14

Implementing Agile Automation

Shadi Mousa

QA Automation Director - Galil Software

1


Automation in Agile – what’s different?

So, where to begin?

Developing Automation in an Agile team

The Automation team & responsibilities in Agile

Agile Automation in large scale organizations

The ATDD/BDD approach

Case Studies

Agenda


Agile requires “fast” automation development and execution; within

sprint time frames.

Collaboration in Agile teams ensures the developed products are

testable by automation tools.

Stories to automate are “simple” and clear – it is easier to prioritize

automation tasks.

Automation milestones should be aligned with product release

milestones.

Automation code refactoring may be needed several times;

especially with immature products and when automating fast.

ROI of automation in Agile is clearer and more measurable.

Automation in Agile – What’s different?


Assess current available automation

Used tools

Developed code and available auto tests

Required automation skills to support current automation

Select appropriate tools and framework

Define automation targets

Automating procedures - continuous environments; development, integration and deployment

Automating installation and configuration scripts

Automating test scripts - regression & progression

Build an automation resources strategy

Who will develop automation code?

Who will develop script level?

Who will lead the automation roadmap, integration, E2E …?

Consult with Galil Software automation experts

So, where to begin?


Developing automation in Agile


Option 1 – Self contained team

Tester skills should include the capability to create the required stories in the automation tool format

Product developers are responsible for implementing the code layer of the automation tool

All required tools and automation framework are the team’s responsibility

Option 2 – Dedicated Automation resources

The larger and more complex the product becomes, the more the need to establish a dedicated team

Automation must include automation experts, including skills with the selected tool

The Automation team & responsibilities in Agile

Role Responsible

Option 1 Option 2

Scru

m T

eam

Test planner Manual Tester Manual Tester

Automation developer Developer Auto Expert Automation executer Manual tester Manual Tester/ Auto Expert

Automation Maintenance Developer Auto Expert …

Au

tom

atio

n

Team

Automation Road Map responsible Scrum Master/Product Owner Automation TL

Infrastructure & Tools ??? Auto Experts Integration Between Multiple Teams ??? Automation TL


Automation in large scale organizations

A Core Automation Team is required

Centralized automation development and maintenance

Responsible for integration and E2E automation

Dynamic automation resource assignment


ATDD – Acceptance Test-Driven Development

BDD – Behavior-Driven Development

The main idea behind ATDD and BDD, in the aspect of automation,

is that the test automation is built before coding, and delivering the

new code depends on the success of the automation

Human language as a base for automation scripts:

User story

“As a [role] I want [feature] so that [benefit]”

Acceptance criteria

Given [initial context], when [event occurs], then [ensures outcome]

Tools that implement ATDD / BDD: JBehave, Cucumber, Robot …

The ATDD/BDD Approach


Product – Operations Management & Optimizer Service

Operations Admin portal – Web Client

Operations Optimization Server – ASP .NET server

Database - MSSQL

Team and Method

Method: Scrum

Team: 5 Developers, 2 QA, 1 Team Leader (Scrum Master)

Team Responsibility

CR, bug fixes, features and improvement requests within a production environment

Sprint Description:

Sprint duration – 3 weeks

Content - contains sub tasks of CR, feature development, design and testing tasks…

Case Study 1


Scrum Master requested automation:

“I want to have a strong regression testing suite that will verify the

correctness of complicated features; this will be used during the sprint

and also in the future when working on other sprints/features.

In addition, the automation should run in the integration environment

before deployment to customer.”

Selected feature to automate:

New optimization option added and activated via a button from the

Administrator Portal; the data entered by the administrator is optimized

upon request.

Automation “Starting point”:

The customer worked with an outdated and unmaintained automation.

The decision was to start from scratch.

Case Study 1 – Cont.


Selected tool – Robot Framework with Selenium2Library

Resources:

The project started with 1 QA engineer who was part of the team

QA engineer had the relevant programming skills

Task assignment as part of the sprint tasks


Defined Script:

Admin enters “Operation” data via Admin portal web application

Activate the new optimization option on the added data

Verify the entered data in the appropriate table in the DB

Verify the converted data in the targeted table is equal to the new table

Run on different sets of data and expected results


Case Study 1 – Cont. Robot Script example *** Test Cases *** TestCase 1 - Login to Admin Portal Open Browser ${AdminPortalUrl} ${BROWSER} Maximize Browser Window ${username} DataReader.getValue ${Credentials_fileName} AdminPortalUser ${password} DataReader.getValue ${Credentials_fileName} AdminPortalPassword Login Portal ${username} ${password} TestCase 2 - Data Sets Import Verification ${DataFileList} ${length} DataReader.getFileList : FOR ${INDEX} IN RANGE 0 ${length} \ Click Link ${Import_Operation Data_link} \ ${XSL_fileName} DataReader.getFileName ${DataFileList} ${INDEX} \ Load Operation Data File ${XSL_fileName} \ Press Button ${Optimize_Vacation_button} \ Press Button ${Accept_Data Change_button} \ ${DB_connection} Connect to Database ${DB_host} ${DB_user} ${DB_password} \ ${isEqual} Compare DB Table to XSL ${DB_connection} ${Vacation_Opt_table} ${XSL_fileName} \ Assert True ${isEqual} Table Data is not equal between ${DB_host}:${Vacation_Opt_table} and ${XSL_fileName} \ Close Browser [Teardown] Close Browser


Results

Auto script was performed successfully, but in the following sprint.

Various data sets were supplied, and as test data

Script was added to the regression suite (first to be added)

Best Practices and Conclusions

Automation should have dedicated resources, and not rely solely on QA engineers’ “best efforts”; especially in order to align automation deliveries with sprint timeframes

On the other hand, “best efforts” solved wasted time in QA engineer resources

Selecting the right tools is crucial; especially if “start point” of no tools in place

QA documentation was added directly to test scripts

The outcome of the scrum team automation must be shared with other teams

Next steps

Request for dedicated resources; by increasing the QA testers with programming capabilities and they perform manual AND automation testing.

In parallel, create and develop automation keywords for previous developed features

Define and execute a work plan for integration environment



Product – ITIL System

Phone rep client – Windows desktop application (Winforms)

ITIL Servers – farm of ITIL servers

Team and Method

Method: Scrum

Team: 4 Developers (1 remote – near shore), 3 QA (offshore), 1 Team

Leader (Scrum Master)

Team Responsibility

CR, bug fixes, features, and future versions of the client

Sprint Description

Sprint duration – 2-3 weeks

Content - contains sub tasks of CR, feature development, design, unit tests

and testing tasks…

Case Study 2


Scrum Master requested automation:

“I want to have a strong regression testing suite that will verify the correctness of complicated features.”

“Automation will be added to the nightly build environment.”

“I also need automation for the client as an independent component without the need to set up the entire system.”

Selected feature to automate:

New field was added to the client; a value entered in the field is sent to the server

Server may reply with different results according to submitted value

Connection timeout should be handled

Automation “Starting point”:

Client – No automation available

Servers – Available APIs



Selected tool – Coded UI (TFS)

Resources:

QA testers in the scrum team had no development skills

Development engineers had no capacity to handle automation

External automation expert was assigned to the scrum team


Defined Script:

Phone rep enters a value in the

client and submits

Rerun over list of optional values

and expected server replies

Simulate reply timeout


Results

Task took longer than expected:

Desktop application was not ready for automation; UI controls missed IDs –

task for development was added to the sprint to reorganize and fix the UI

controls IDs

Auto script was performed successfully, yet completed in the following

sprint

Script was added to the regression suite

Isolating desktop application was performed after writing the script


Isolating desktop application was

performed by simulating server

connection and responses on local

port of the automation client


Best Practices and Conclusions:

Team Leader feedback:

Adding the automation resource to the scrum team and participating in the design made the application better designed and implemented

Automation should have dedicated resources, with the relevant skills

A supportive team for automation is required, since in such a large system, self contained automation resources cannot perform all required automation tasks

There is a need for automating utility / configuration scripts such as upgrading environments…

Next steps:

To add continuous tasks for developers to make the application testable by exposing the appropriate control IDs, recognized by automation tool

Define and execute a work plan for continuous integration environment



Thank You

Shadi Mousa

QA Automation Director - Galil Software