Agile Test Automation - WordPress.com · Agile Test Automation Key Requirements for Functional and Performance Testing Introduction Agile test automation may sound like a contradiction

Agile Test Automation

Key Requirements for Functional and Performance Testing

Linda Hayes, Founder, Worksoft Inc.Shoeb Javed, Vice President of Engineering, Worksoft Inc.

© 2010 Worksoft Inc. All rights reserved. Worksoft and Worksoft Certify are registered. All other trademarks are properties of their respective owners.

PG. 2

Agile Test Automation Key Requirements for Functional and Performance Testing

Contents

Executive Summary/Intro. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Continuous Integration Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

User Interface Prototyping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Agile Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Developing Functional Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Updating Functional Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Expanding Test Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Maintaining Functional Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Executing Functional Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Certify Testing Certify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

About the Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

About Worksoft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11


PG. 3


Introduction

Agile test automation may sound like a contradiction in terms, but it is a new testing methodology that can revolutionize the world of application testing, bringing with it order of magnitude gains in time, cost and risk management.

Traditional automated testing methods, such as record/replay tools, provide a useful, but rigid approach for application testing. These tools generate custom script code that must be developed and maintained along with the application under test. Conventional wisdom holds that these tests should not be automated until the application is stable, resulting in a primary focus on regression tests.

In general, traditional record/replay tools do not lend themselves well to automating tests in a rapidly changing environment. The cost of maintaining test scripts for an application with a high rate of change would exceed the benefit gained from automated execution. However, a newly developed methodology—Agile Test Automation—is providing more flexible test management strategies.

There are two aspects of any agile development process that make it ideal for automation testing if only the right tool was available. These are:

Continuous Integration Environment

Any agile development methodology requires a continuous integration environment. Designed to ensure that each software component is functional, a continuous integration environment runs unit tests to ensure the system is compiled and built successfully. The drawback is that this method doesn’t test the system holistically to ensure that each component functions in relation to every other component. But, what if defining and executing automated functional tests was easier and faster than writing and running unit tests? You would know whether your product worked within moments after you make and check in a change!

User Interface Prototyping

Most agile development projects have methods for rapidly prototyping end user functionality to keep focus on end user requirements rather than abstract design. What if


PG. 4


there was a way to quickly write functional tests against a user interface prototype and have these tests transfer with minimal modifications to the finally constructed product? You would be assured that the product you delivered met the expectations of the users it was designed for. What if you could write your test against one technology platform and then run the same tests against another totally different one?

The radical code-free design of Worksoft Certify® changes everything. It not only allows tests to be developed before the code, it also enables tests to be maintained automatically as the application changes. This ensures tests written against a user interface prototype can be converted to run against the real application. Not only does this new approach enable test automation to be agile, it further accelerates the agile development process by reducing cycle time for testing. Read on to discover how Worksoft Certify makes agile test automation a reality.

Agile Development

Agile development approaches are enjoying acceptance by companies seeking to reduce cycle times while improving quality. There are many flavors of agile development, but all focus on short interval deliveries using a cross-functional team. While all 12 key practices that define agile development methodologies contribute to the end result, the “integrate early, integrate often” and “test-first” philosophies are intended to keep both programmers and users focused on validation throughout the process.

As applications develop in sophistication and complexity over time, testing efforts expand considerably given that development iterations are cumulative. Without an automated approach, testing can quickly become a bottleneck as each successive iteration contains greater functionality to be tested and therefore, more complexity.

Automating unit testing during development is relatively straightforward for programmers, since they can manually augment their code and take advantage of tools that exercise testing and diagnostic components directly. Developing automated functional tests is more challenging, since traditional tools rely on a recording or scripting paradigm. These existing methods assume the software is already available. Further, they require tests to be rewritten when rapid changes are made to the application as is typical in an agile environment.


PG. 5


Using Worksoft Certify to automate functional tests introduces a dramatic departure. There is no recording or scripting. It is designed specifically to allow expert users to develop their functional tests in parallel with the code, so that ambiguity about expected results is removed and there is minimal delay between delivery of a testable component and its validation. Not only does Certify enable both application and automated test development to proceed concurrently, it further contributes to overall quality by enforcing standard naming conventions that are a hallmark of agile coding standards.

Worksoft Certify itself is also agile in that it easily accommodates changes to both application and functional tests, so that the fluidity of the development process is not constrained by the rigidity of the test approach. This white paper describes Worksoft Certify and its approach to agile automation. It will explain how Certify can be used to help your team fully realize its benefits while embracing an agile development methodology.

Requirements

The typical iteration in agile development starts with requirements or stories: not unlike use cases, these describe a discrete set of circumstances and user interactions with the system that are testable and can be easily broken down into programmer tasks whose implementation can be estimated. These requirements may be described in narrative text, defined in a modeling tool, or prototyped.

In Certify, requirements are defined in a hierarchical structure that depicts the business processes to be supported by the application and the variations of each that are triggered by business rule outcomes. These requirements are then linked to processes. A process is a series of steps that execute a business process or task, providing input and verifying results.

Certify processes can be identified as soon as the requirements are. While capturing requirements on paper may appear to be the easiest technique, entering them into Certify has the added benefit of storing them in a repository that is easily shared across the entire team. Furthermore, as development progresses, successive levels of detail can be added to the processes until they are executable test cases.


PG. 6


Specifications

Once a requirement is defined, it is broken down into tasks for the developer, and the portions of the application that interact with the user must be specified either on paper or as a prototype. This specification is approved by the users and enables both the development and documentation processes to commence.

For test purposes, the requirement must be broken down into the steps that perform validation. In Certify, these steps are developed using actions against the objects within the application. The objects are described in an application map, which is an inventory of components including windows, screens, pages or messages and the fields or elements they contain. This map can be created within Certify or imported from a prototype metadata model.

Agile development calls for coding conventions so that all code is uniformly formatted and classes and methods are named the same. This is important as the code belongs to the whole team, and should look familiar no matter who wrote it. What’s interesting is that there is less of an emphasis on naming windows and objects consistently, which is an essential requirement for repeatable functional test automation.

For example, a product code could be entered as a straight text field, a combo box, or as a list. The benefit of a combo or list box is that it prevents the user from making mistakes by entering an invalid product code, but depending on how many product codes there are, a combo box may be too unwieldy. The programmer may choose a combo box without knowing that there are potentially hundreds of product codes, or choose a list when there are only a few possible choices. User participation prevents these types of design errors from the outset.

But no matter, Certify can help. The Certify application map serves as a dictionary by giving each and every object a name that is unique and persistent from build to build. The map also describes the class or type of object to be defined. This is an important and powerful capability, because it allows the user to participate in discussions about how the object should be implemented.

Once the requirements are identified and the map has been created, Certify has everything it needs to empower the user to develop fully documented, automated functional tests.


PG. 7


Developing Functional Tests

Certify combines the documentation and automation of tests into one simple step. All the user has to do is point and click their way through selecting the window(s) the test process will traverse, the objects to interact with, the actions to take such as entering data or verifying responses, and how the test should respond based on the result.

What makes Certify functional tests especially powerful is that they can be printed in an easy to read, narrative format that everyone can review and understand. In fact, it’s amazing how often just reading through the step-by-step of a story reveals illogical assumptions or awkward workflows. These narratives can be provided to developers as detailed tests and to technical writers as useful documentation of user scenarios.

Once developed, these tests can be executed – automatically – against the software as soon as it is available. This reduces cycle time by providing rapid feedback on each build. Further, as functionality continues to expand during successive iterations, the corresponding repository of tests also grows, assuring full coverage so that the continuous integration process does not compromise preceding capabilities.

Updating Functional Tests

One of the advantages of agile development is that it is change-friendly. Short iterations give developers rapid user feedback. In traditional test environments, this may create resistance to automating functional tests too soon as constant changes require too much effort to rewrite or maintain.

But Certify makes changes easy. Simply modify the map and change the steps that make up the processes with simple insert, delete, and cut and paste functions. Any time lost in keeping tests updated is more than offset by the benefit of reducing tests to a common structure and format in a repository that can be shared throughout the cross-functional team.

Expanding Test Coverage

Once a requirement is documented and the steps are well understood, Certify allows the user to easily expand test coverage by quickly adding data variations. By substituting


PG. 8


variable names for data constants and attaching rows of data values to a process, a user can convert a single test into dozens, hundreds or even thousands. This capability assures that the functional tests have the widest possible coverage.

Maintaining Functional Tests

As the code is completed and testable components become available, Certify allows the application map to be captured directly from the software or metadata. This is a critical capability because it will instantly identify any inconsistencies in naming conventions for windows or object, or differences in the class or type of object that was implemented.

Not only will Certify find the differences, it will instantly provide an impact analysis by showing all processes and steps that are affected by the change. This is an easy, automated way of ensuring that all naming conventions were followed and that the objects behave as expected. If discrepancies are found and the user agrees to waive them, Certify allows a search and replace facility to substitute the changes and update the affected processes.

Executing Functional Tests

Once the testable components are delivered and all object names are verified and reconciled if needed, tests can be executed. Depending on how much functionality is ready, individual processes can be grouped into suites that exercise robust, end to end processes. This bottom-up approach lets users start with individual requirements and build their way up into a complete system.

Test execution may also be tied into the daily or continuous build infrastructure so that automated functional tests are executed each time code is checked in or each night during the nightly build process.

Certify provides a detailed, step-by-step log that is a mirror image of the process steps, with actual and expected results added. Any errors or unexpected results result in a screen capture so the each situation is documented as it occurs. And Certify captures even more than functional validation. Each and every step in the log has an elapsed time measurement, as does the entire process. If the software meets its functionality goals, but the response time is unacceptable, Certify will let you know.


PG. 9


What Certify accomplishes is to transform the tasks that drive the development iteration into automatically executable Certify processes that validate whether the delivered software meets the specified requirements.

Certify Testing Certify

Certify itself is developed using an agile development process and provides a real world example of integrating automation testing into a continuous build environment.

The figure below shows a simplified process flow of how Certify is used to test itself in a continuous integration environment.

Source Control

Dashboard BPP

Virtual Machine - 1

Virtual Machine - 2

Virtual Machine - n

Cruise-ControlBuild Service

Certify Connect Web Services

Certify Connect Server

Certify Database Server

Certify Virtual Machines

BuildServer

A build is triggered by file check-in or scheduled nightly.

Successful completion of build & unit tests triggers Certify virtual machines to install new software.

Certify functional tests (from the upsteam Certify database) are executed on the new build.

Results notification is sent via email to the build team after functional test execution is complete.

Certify Dashboard is used to analyze test results, requirements coverage and project progress. Certify BPP is used for documentation.

1.

2. 3.

4.

5.

UpstreamCertify

DownstreamCertify

CertifyConnect


PG. 10


Certify developers are responsible for building functional tests for new functionality that they are developing. Virtual machines are set up with Certify pre-installed and pointed to a database (the upstream database) of functional tests that are continuously augmented by the developers. Integration builds are triggered by software check-in or through a timed nightly build. On each successful build completion, the Certify virtual machines pull down the new version of Certify and run the functional tests (this process is very fast since tests are automated and does not add significant overhead to the build process). Results of the test run are then emailed to the build team for review.

The build team can optionally use the Certify Dashboard to further analyze the results, requirements coverage, project progress, etc. In addition the Certify Business Process Procedure module can be used to generate documentation that describes how a feature is supposed to work. This is done using the self documentation narratives generated by Certify when a functional test is executed. This entire process can be run several times a day to ensure that the system is always functional in an ever changing environment.

Summary

Agile methodologies hold out the promise of rapid development without sacrificing either functionality or quality. Worksoft Certify enables functional testing to both guide and control the entire development process within the context of an orderly, structured approach that meets agile development goals of consistency and rapid turnaround. Its ease of use overcomes the usual objections to investing too much too soon, and its automated maintenance capabilities address the potential for downstream changes to affect the functional tests.

Certify customers have reported dramatic time and cost savings based on its fully concurrent, agile approach.

About the Authors

Shoeb Javed is Vice President of Engineering at Worksoft. Shoeb is an experienced executive with over 17 years of experience leading the development of cutting edge network infrastructure, enterprise software and service management solutions. Prior to Worksoft, Shoeb was CTO of Variview Technology where he was responsible for product


PG. 11


development, operations and business development. Shoeb has also held leadership roles in companies like Ericsson, M68 technologies, Vesta Broadband Services and Intelect Network Technologies. He has a master’s degree in Electrical Engineering from the University of Hawaii and a Bachelors degree in Electronics Engineering from the University of Nagpur, India.

Linda G. Hayes has over 20 years of experience in software test automation and is the founder of three software companies including AutoTester, the first PC-based test automation tool, and Worksoft, Inc., pioneer of next generation code-free test solutions. Linda holds degrees in accounting, tax and law and is a frequent industry speaker and award-winning author on software quality. She has been named as one of Fortune Magazine’s People to Watch and is a regular columnist and contributor to www.StickyMinds.com and Better Software magazine, as well as a columnist for Computerworld and Datamation and the author of the Automated Testing Handbook.

About Worksoft

Worksoft develops innovative and comprehensive solutions to automate lifecycle management for enterprise applications. Worksoft solutions are designed to meet the specific needs of both business and technical users, thus enabling complete, end-to-end business process validation. Worksoft customers gain better visibility into the impact of business process changes, improve application test coverage, and realize faster deployment times for every type of change. Worksoft solutions are specifically designed to meet the unique architectural needs of SAP application environments and the specific needs of both business and technical users. Worksoft customers realize faster deployment for End-to-End Business Process Validation times related to upgrades of SAP solutions, gain better visibility into the impact of business process changes, and further increase performance and availability.

Founded in 1998 and headquartered in Dallas, Worksoft is privately held by Austin Ventures and Crescendo Ventures. Worksoft customers include Fortune 500 companies across manufacturing, financial services, government, healthcare, retail and transportation.

For more information on Agile Test Automation or Worksoft Application Lifecycle Management solutions, please contact Worksoft at 1.866.836.1773 or visit www.worksoft.com.

© 2010 Worksoft Inc. All rights reserved. Worksoft and Worksoft Certify are registered. All other trademarks are properties of their respective owners. www.worksoft.com

Documents

Agile Test Automation - WordPress.com · Agile Test Automation Key Requirements for Functional and Performance Testing Introduction Agile test automation may sound like a contradiction