Building an Automation Framework around Open Source Technologies

STC 2009- Building an Automation Framework around Open Source Technologies - 1 -

MindTree Limited

mindtree Page 1 11/12/2009- 1 -

Title - Building an Automation Framework around Open Source Technologies

Paper : Building an Automation Framework around Open Source Technologies

Conference : STC 2009

Author : Muralidhar Yalla

E-mail ID : [email protected]

Address : MindTree Ltd.

Maruthi Chambers, No. 17/9C, Rupena Agrahara,

Hosur Road, Bangalore - 560 068,

INDIA

mailto:[email protected]


MindTree Limited

mindtree Page 2 11/12/2009- 2 -

1. Abstract

Re-usable test automation frameworks coupled with open-source tools and technologies is a key

solution to shrink test cycle times and related costs. This paper covers an approach for creating a script-

less web application automation framework around SeleniumRC which is robust, flexible, and

extensible. Our experimental results on this framework show increased productivity and ROI, reduced

learning curve and dependency on skilled resources, eliminated usage of commercial tools and manual

configuration. The framework provides ease of script generation & management, concurrent and

remote execution, test data generation and customized reports with screenshots. Preliminary statistics

indicate that the automation framework reduces ~ 40% of total test execution effort per release cycle

under a given scenario.

2.Introduction

Frameworks are built at each level of the testing phase/cycle viz. unit, integration, and system testing.

Each framework is built on a set of rules which are derived based on need analysis.

This paper describes the methodology used in developing a robust web based automation framework

and supported by a case study on how this framework is tuned for ease of use, robust and scalable

framework.

What are the guidelines in building a framework on top of open source test automation technologies?

We need to perform a detailed automation need analysis study, to evaluate the scope and expectations

of the automation framework to be developed. The need analysis is an extensive checklist of generic

questions like “Supported browsers IE6+, FF2+, Safari, Supported platforms, protocols“ etc., which

will enable us to take a decision on go, no-go for developing an independent test automation

framework.

Framework / Automation Objectives: Identify the basic and must have features to be supported by

the framework, which will be the result of your detailed need analysis that gives us a broad wish list of

test automation objectives:

• Powerful framework for organizing test design, automation, execution around keywords, test

report generation and test data generation

• Highly Productive approach of Test Script Generation from English like language organized

as user actions, objects and values

• Framework should offer Library of Reusable Functions, like color validation, image

comparison, file Upload etc., which are extensible

• Easy to use and maintain Framework scripts by business users

• Generates test results in html format consisting of test summary, detailed reporting and with

screenshots.

• Supports component based model for object repository

• Supports Remote and Concurrent execution, which will reduce the total execution time


MindTree Limited

mindtree Page 3 11/12/2009- 3 -


MindTree Limited

mindtree Page 4 11/12/2009- 4 -

How should I start to identify the best open source tools / frameworks for my project?

This is a common question that customers / project teams ask. To accomplish the automation

objectives, the test managers / framework developers should adhere to the best practices as sited below

• Identify categorically the nature of application under test say, thin client / thick client / web

Services etc.,

• Current scope of this framework is considered for a thin client (Web application) automation.

• Prepare a checklist of features that the open source tool should support, ensure that the

automation objectives met or >~50%.

• Identify the open source / freeware tools that best suits your automation objective, say

Selenium-Grid / TestNG for creating concurrent test script execution.

• Perform a functional automation tool evaluation for details refer table1.1, this mechanism for

test automation projects is to correlate, identify the best fit single / multiple combinations of

open source tools that maps your automation objectives.

Figure 1: Sample data depicting the tool evaluation process

Figure 2: Instructions to fill Figure 1 template with real weighted scores

Tool evaluation provides a comprehensive overall-weighted

score, with which we can clearly identify the best possible

combination and as a best practice we ensure to architect the

framework with a combination of open source tools.

MindTree has leveraged its years of automation

expertise with open source technologies to create effective

but easy to use automation framework called Selenium

Automation Framework (SAF). Figure 3: High level architecture - SAF

MindTree Selenium Automation Framework (Figure 3) has strengths of SeleniumRC, TestNG,

Java, ReportNG and ANT. This combination ensures that the framework developed meets the

automation objectives.


MindTree Limited

mindtree Page 5 11/12/2009- 5 -

3.SAF Architecture and Design Approach

This paper describes the “Plug and Play” hybrid framework architecture on top of open source tools; as

a best practice this architecture ensure that the

advantages of the multiple tools can be leveraged.

As per the current design and implementation

shown in Figure 4, this framework is scalable

because the controller and the execution scripts are

designed as individual modules / utilities which

give extensibility and ease of maintenance provides

the best scalable solution to run tests in parallel Figure 4: Work Flow-Module level architecture

How do we create a “Plug and Play” framework that best suit my automation objectives, project needs?

Basic components of an automation framework consist of controller, object repository, test data,

execution engine, reporting and error logging and recovery.

• There are three broad approaches in designing a controller. Spreadsheet driven, XML driven

and database driven.

• Spreadsheet driven is proven for its ease of use by manual test engineers and XML driven is

proven for its extensibility, say supports web services testing and database driven is proven for

information security, say automation of Financial projects

• Controller is the core of the framework which is used to govern the execution work flow and

modules listed below

• Configuration Module: This module will load the required test script execution environment to

govern the execution work flow

• Test Data Module: Test data stored in

spreadsheet format (Figure 5) and this will

provide input for the automation test scripts. Figure 5: Sample Test Data

• Object Repository Module: Web elements / objects stored in spreadsheet format (Figure 6) and

this will provide the same as input to the retailer test scripts

• Libraries: This is used to load generic and

application specific methods, reusable

components grouped as a method calls in

the spreadsheet as shown.

• Logging Module: This module manages

logging across the work flow, Log4j can be used for debugging and error handling.

Figure 6: User Defined Methods Template


MindTree Limited

mindtree Page 6 11/12/2009- 6 -

• Reporting Module: Custom html reports with screenshots, ReportNG is used which is a plug-

in to TestNG framework

SAF is used for web application testing. The above architecture was chosen as it catered to most of the

core requirements, such as a hybrid driven framework addressing concurrent test script execution,

multiple browser support, extensible, easy to use, data driven testing, execute only failed or passed test

cases and with good reporting. SAF is driven by a simple spreadsheet where the application

configuration, test data, test scripts and other inputs required for the execution of the complete

automation cycle are achieved in simple steps:

• Environment setup

• Develop Test cases

• Master test suite Configuration

• Execute tests

• Report Results

Figure 7: Layered view - SAF

Figure 7 – SAF Layers explains integration of SAF with open source technologies and how each of

these technologies interlinked, revealing the potential of SAF. It contains the following components:

Test environment initialization – Plug and Play design

Test execution engine – Controller driven

Error management and recovery

Status and reporting engine

Test Environment InitializationThis module will set all the required parameters for a test run. The central framework engine will read

the global configuration file and will set the environment required for the application under test (AUT).

SAF user has the flexibility to update the configuration file to select and execute tests by customizing

the following parameters (Figure 8):

Base URL of the AUT Type of

Browser

Remote or Local machine

Single or Multiple machines

Test Suite execution speed - fast /

moderate / slow and

Remote test case execution

monitor, which helps capture

intermediate screenshots

Figure 8: Sample Test Suite Configuration

SAF controller / execution engine will start the Selenium-RC server and confirm the status of server

back to the Test Execution engine.


MindTree Limited

mindtree Page 7 11/12/2009- 7 -

Test Execution Engine

Execution engine clearly separates test case design and test framework execution. SAF provides an

easy way to organize test cases and test data. The test execution engine uses ANT to consolidate the

targets and execute them sequentially from starting the selenium server and then running the TestNG

suite file which will execute the test cases in the excel sheet and then shut down the selenium server.

MS Excel was chosen given its ease of use and its capability to choose keywords from the list of drop

down and check boxes. User has the liberty to execute a single step or multiple steps in a test case or

single test suite or multiple test suites.

The test design is performed, based on keywords, web objects, test data and a flag to exclude or include

the test steps / tests cases.

Figure 9 shows a sample test case to launch the application browser, performs login and user actions.

Here one single excel sheet can consists multiple test scripts

Figure 9: Sample test cases filled spreadsheet

Steps to fill the sheet are as given below:

Write the test case id under TCID.

Select Action “start” from the dropdown in Excel to specify the start of test case.

Select the open, type, click action as required

Use Selenium-IDE, Xpather, FireBug, DOM Inspector to identify the web element ID’s to fill

the Object columns accordingly.

Select End under action to specify end of the test cases.

Select Stop under action to specify end of the test suite execution.

In the above steps for designing test cases, the process of capturing the web elements using DOM or

Xpath and validating the same for all the intended browsers is time consuming, but this would be a

onetime effort. and as we use excel sheet for keywords / web objects the maintainability of the test

cases is comparatively very low for the sub-sequent QA cycles.


MindTree Limited

mindtree Page 8 11/12/2009- 8 -

If the web application under test is undergoing frequent change, SAF user has to update web

components identified web ID/Xpath/DOM in a separate Object Repository spreadsheet in Figure 10

and refer the user defined custom name for that object in all his

scripts this will help to maintain global object repository per

page / module which will help manage/update scripts at one

single place instead of all the individual test scripts. Figure 10: Sample Object Repository

SAF test execution engine helps us to execute the test cases in parallel or in sequence using TestNG

framework and will return the test case execution results in html, email report.

Error Handling and Recovery Engine

The Test Execution Engine helps in recovery process. It would invoke the application from existing

state and continue test execution from the last aborted test script. TestNG on the other side will help

execute only the failed test cases in a test suite and thereby reducing the verification time. Selenium RC

and TestNG have mechanisms to capture the error logs. In case of a failure or crash, the error log will

be sent as an email along with the test case execution report.

Reporting Engine

The default report of TestNG is comprehensive, but it’s not easy to understand at-a-glance; as a result

SAF is integrated with a ReportNG listener which is a simple HTML reporting plug-in for the TestNG

framework. After the test run is completed an email alert will be sent to the stake holders.

Figure 11: SAF - Test Results Summary Report

Figure 12: Report showcasing Data Driven Test Results

http://www.testng.org/


MindTree Limited

mindtree Page 9 11/12/2009- 9 -

4.Case Study

A Performance & Talent Management System Case Study snapshot. This is a web based Employee Performance & Talent Management system that provides companies with superior control over the lifecycle of performance appraisal. The system has several modules like career development planning, total goal management, recruiting etc. The purpose of the case study is to pilot the run of itest Selenium Automation Framework in addressing the challenges listed.

• Testing AJAX components• Testing floating menus and custom web

elements• Testing dynamic popup windows• Time spent in manual test case execution• Browser Compatibility testing • Worked with dynamic pages / objects• Searching a particular user or customer when

logged into application as Super user which contains more than 270 users and in particular no search button option is available.

• Identified 5 complex test scenarios which have all the listed challenges

• SAF hides the underlying complexity in developing test suites in Java / Perl / PHP / Ruby / Python / Selenese programming language

• Thus, SAF could offer fast test case design and a cohesive way of grouping of test cases.

• Given SAF's ease of use, it helped develop complex test cases using simple keyword actions, web objects, test data, and test execution

• In one go we were able to get results in html, excel report generation which used the TestNG framework and sent a detailed report in an email.

• SAF is used to execute part or full test suite like only failed test cases, etc.

• Establishes a Regression Test Automation framework which is flexible, scalable and maintainable – Protection of investments

• Automated regression suite reduces costs, improves quality and makes release process Predictable, Faster and Reliable

• More than 75% productivity improvement using SAF

• SAF is an extensible framework, where future enhancements are just plug-ins to the existing code base and ease to maintain.

• SAF could capture the state of the remote application by capturing the screen shots and the same is sent along with the test repots in an email.

Advantages for the end-users of SAF:• Created a component based test framework

that can be bundled together based on Page Modules

• Automation can be achieved with minimal coding knowledge

• Built over open source tools/ libraries/ frameworks

• Support for sequential and concurrent execution

• Support for remote execution and monitoring

• Selective execution of test scripts• Selective test script generation for expert

view


MindTree Limited

mindtree Page 10 11/12/2009- 10 -

5.Conclusion

Thus we successfully developed a framework that helped us to exceed preset framework/automation

objectives and could execute test scripts concurrently, reducing the execution time from 2 Man days to

60 Minutes, bringing in 75% productivity improvement using SAF on a continuous basis. We believe

this approach can be carried across companies to significantly reduce the automation execution cost

without compromising on Quality. The future work of this initiative is directed towards further

evolution of SAF, SAF-Flash and “SAF as a SaaS” development model.

6.Definitions, Abbreviation and Acronyms

Acronym DescriptionURL Universal Resource LocatorAUT Application Under TestDOM Document Object ModelUDE User Defined ExceptionSAF Selenium Automation FrameworkController Framework Core used in conjunction with execution engine

7.ReferencesItem DescriptionReference1: http://www.openqa.org/selenium/Reference2: http://www.testng.org/

Reference3: http://agiletesting.blogspot.com/2006/01/useful-tools-for- writing- selenium.html

8.Biography of the authors

Author1: Muralidhar Yalla (Murali), Sr. Test Specialist in the Innovation Labs team at MindTree.

He has over 10 years of Software Industry experience working on open source functional automation

testing tools like Selenium, Sahi, Watir frameworks like TestNG, PHPUnit, and others, TCM tools and

code coverage tools for Java applications like EMMA and others with various languages including

Java, Perl, PHP, Ruby, and Python on WinXP.

Earlier to MindTree, Murali was with Yahoo Corporation, for 2 years, where he has worked on a

number of projects involving API testing, web services testing and functional automation using various

languages viz. PHP, Perl, shell and others on FreeBSD. He has previous stints at Synopsys, Zensutra,

Havells and Protel where he has worked on world class client / server products and worked in the area

of application development and testing of Device Drivers, worked with various languages like C, IPC

and others on Red-Hat Linux. He also enjoys working with more dynamic, productive, multitasking

and fun languages.

http://agiletesting.blogspot.com/2006/01/useful-tools-for-writing-selenium.html



http://www.testng.org/

http://www.openqa.org/selenium/


MindTree Limited

mindtree Page 11 11/12/2009- 11 -

Co-Author 2: Ramanath Shanbhag (Ramanath), Program Director at MindTree Ramanath heads

the Innovation Labs, Delivery Excellence and Delivery Assurance activities of the Testing Business

Unit. He has played key roles in business needs analysis; organizational assessments; assisting in

building Test Competency Center, solution accelerators and frameworks; training; and consulting. His

recent contributions have been Testers’ Toolkit, Pricing Models for Outcome based engagements and

Test Quality Index prediction (TQIp) model to name a few. He has exhibited thought leadership

through external and internal evangelism like presenting papers at Step-In Conferences, and conducting

Test Leadership Programs within the organization. He is an active member of Technology Solutions

Group, and is involved in proposal creation and pre-sales presentations. He also works towards

promoting organic growth in-line with the company’s competency framework through effective

training and talent transformation programs. He has expertise in incubating complex projects, and

growing those engagements into large teams through delivery excellence and value-additions. He has

been successful in growing many projects into Multi-Million Dollar engagements. Prior to joining

MindTree Ltd., he has worked with prestigious companies like Aditi Technologies, Microsoft

Corporation, and Accord Software (a consultant to HP).

9.Appendix

Some core definitions

• TestNG is a testing framework inspired from JUnit and NUnit. It introduces powerful and

easier to use functionality. The framework covers all categories of tests: unit, functional, end-

to-end, integration, etc

• Selenium is an open source functional and acceptance testing tool for Web applications. It

supports multiple browsers, platforms, complex AJAX-based web user interfaces under a

Continuous Integration system. It facilitates users to write their own tests in java / PHP / Perl /

Ruby / Selenese / Python.

• DOM Inspector /Firebug is a browser plug-in, used to access the live DOM of web

application under test.

• ReportNG is a HTML test results reporting plug-in for TestNG framework

https://reportng.dev.java.net/

https://addons.mozilla.org/en-US/firefox/addon/6622

http://selenium-rc.openqa.org/

http://testng.org/

Documents

Building an Automation Framework around Open Source Technologies