Software Test Automation in Agile Projects - Keytorc Approach

Software Test Automation in

Agile Projects

Agenda

» Test Automation Basics

» Continuous Integration/Delivery in Nutshell

» Stages of CD Pipeline

» Best Practices – Tools & Patterns

» Next Generation Approaches

Test Automation Basics

Test Automation

The major key to the success of software test automation:

To reduce manual testing activities and redundant test operations using a systematic

solution to achieve a better testing coverage.

Test automation is not always necessary,

appropriate, or cost effective. In cases

where we are making decisions

based upon an expected return on

investment, analysis can direct us to

where test automation can benefit us.

These returns are best computed by

comparing the costs and gains achieved

through test automation over

manual testing.

Automation Decision in Sequential Models

Frequent regression testing

Repeated test case execution is required

Faster feedback to the developers

Reduce the human effort

Test same application on multiple environments

Pyramid of Test Automation

Business Facing

Technology Facing

Continuous Integration/Delivery

İn Nutshell

Definition of Continuous Integration

Continuous Integration is a software development practice where members of a team

integrate their work frequently, usually each person integrates at least daily - leading to

multiple integrations per day. Each integration is verified by an automated build

(including test) to detect integration errors as quickly as possible.

(Martin Fowler – Thoughtworks)

Continuous Integration Practices

Maintain a Single Source Repository

Automate the Build

Make Your Build Self-Testing

Everyone Commits To the Mainline Every Day

Every Commit Should Build the Mainline on an Integration Machine

Fix Broken Builds Immediately

Keep the Build Fast

Test in a Clone of the Production Environment

Make it Easy for Anyone to Get the Latest Executable

Everyone can see what's happening

Automate Deployment

A Basic Pipeline

Continuous Delivery

What to Automate

Stages of

Continuos Delivery Pipeline

An Example Toolset

Testing Phase: Commit Stage

CommitStage

AutomatedAcceptance

Stage

Non-FunctionalTest Stage

Manual Tests

SanityCheck

Rather be a scheduled

job

Commit Stage

• Run against every check-in

• If it fails, fix it immediately

• Creates a release candidate

Hint: If next stage(s) (Automated Acceptance Tests) take a significant amount of time

(e.g. More than 30 minutes), embed a small subset of them into commit stage. So,

feedback interval will be decreased to act fast on major incidents.

Testing Phase: Automated Acceptance Stage

CommitStage

AutomatedAcceptance

Stage

Non-FunctionalTest Stage

Manual Tests

Sanity Check

Automated Acceptance Stage

Tips:

Focus on multi-threading for race conditions

Run tests in parallel (Use a supporting framework)

Design atomic scenarios

Manage data strictly

• End-to-end tests in production-like environment

• If it fails, fix it immediately triggered when upstream stage passes

Testing Phase: Non-Functional Test Stage

CommitStage

AutomatedAcceptance

Stage

Non-FunctionalTest Stage

Manual Tests

Sanity Check

Non-Functional Test Stage (Performance)

• Select most suitable test cases (Either widely used or critical for a business)

• Test against a production replica environment, for example staging (As much

as possible)

• Focus on subject matter practices (Anything!)

• Use automate-acceptance tests with counters (As a first step maybe)

Testing Phase: Manual Tests

CommitStage

AutomatedAcceptance

Stage

Non-FunctionalTest Stage

Manual Tests

Sanity Check

Manual Tests

• Focus on non-repetative, unautomated test cases, showcases

• Create room for creativity with exploratory testing

• Realize people dependent tests, like usability

Pipeline Integrated

Pipeline Integrated

Pipeline Patterns & AntiPatterns

Insufficient Parallelization

Ideal Time < 10-15 Mins

No more than 1 hour

Hint: Prefer a paralellizable unit testing framework. TestNG for Java and MbUnit for .Net

might be good choices.

Pipeline Patterns & AntiPatterns

Inflexible Workflow

References

http://www.romaniatesting.ro/to-automate-or-not-to-automate-testing

http://www.softwarequalitymethods.com/papers/star99%20model%20paper.pdf

http://www.continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/

http://www.testng.org/doc/documentation-main.html#parallel-tests

Referencing Lisa Crispin, Agile Testing Book 2009

Continuous Delivery Workshop Material created by Jez Humble, Martin Fowler, Tom Sulston, & Neal Ford

Teşekkürler

info@keytorc.com

Keytorc

Keytorc Software Testing Services