Embed Size (px)
Citation preview
Software Test Automation in
Agile Projects
Agenda
» Test Automation Basics
» Continuous Integration/Delivery in Nutshell
» Stages of CD Pipeline
» Best Practices – Tools & Patterns
» Next Generation Approaches
Test Automation Basics
Test Automation
The major key to the success of software test automation:
To reduce manual testing activities and redundant test operations using a systematic
solution to achieve a better testing coverage.
Test automation is not always necessary,
appropriate, or cost effective. In cases
where we are making decisions
based upon an expected return on
investment, analysis can direct us to
where test automation can benefit us.
These returns are best computed by
comparing the costs and gains achieved
through test automation over
manual testing.
Automation Decision in Sequential Models
Frequent regression testing
Repeated test case execution is required
Faster feedback to the developers
Reduce the human effort
Test same application on multiple environments
Pyramid of Test Automation
Business Facing
Technology Facing
Continuous Integration/Delivery
İn Nutshell
Definition of Continuous Integration
Continuous Integration is a software development practice where members of a team
integrate their work frequently, usually each person integrates at least daily - leading to
multiple integrations per day. Each integration is verified by an automated build
(including test) to detect integration errors as quickly as possible.
(Martin Fowler – Thoughtworks)
Continuous Integration Practices
Maintain a Single Source Repository
Automate the Build
Make Your Build Self-Testing
Everyone Commits To the Mainline Every Day
Every Commit Should Build the Mainline on an Integration Machine
Fix Broken Builds Immediately
Keep the Build Fast
Test in a Clone of the Production Environment
Make it Easy for Anyone to Get the Latest Executable
Everyone can see what's happening
Automate Deployment
A Basic Pipeline
Continuous Delivery
What to Automate
Stages of
Continuos Delivery Pipeline
An Example Toolset
Testing Phase: Commit Stage
CommitStage
AutomatedAcceptance
Stage
Non-FunctionalTest Stage
Manual Tests
SanityCheck
Rather be a scheduled
job
Commit Stage
• Run against every check-in
• If it fails, fix it immediately
• Creates a release candidate
Hint: If next stage(s) (Automated Acceptance Tests) take a significant amount of time
(e.g. More than 30 minutes), embed a small subset of them into commit stage. So,
feedback interval will be decreased to act fast on major incidents.
Testing Phase: Automated Acceptance Stage
CommitStage
AutomatedAcceptance
Stage
Non-FunctionalTest Stage
Manual Tests
Sanity Check
Automated Acceptance Stage
Tips:
Focus on multi-threading for race conditions
Run tests in parallel (Use a supporting framework)
Design atomic scenarios
Manage data strictly
• End-to-end tests in production-like environment
• If it fails, fix it immediately triggered when upstream stage passes
Testing Phase: Non-Functional Test Stage
CommitStage
AutomatedAcceptance
Stage
Non-FunctionalTest Stage
Manual Tests
Sanity Check
Non-Functional Test Stage (Performance)
• Select most suitable test cases (Either widely used or critical for a business)
• Test against a production replica environment, for example staging (As much
as possible)
• Focus on subject matter practices (Anything!)
• Use automate-acceptance tests with counters (As a first step maybe)
Testing Phase: Manual Tests
CommitStage
AutomatedAcceptance
Stage
Non-FunctionalTest Stage
Manual Tests
Sanity Check
Manual Tests
• Focus on non-repetative, unautomated test cases, showcases
• Create room for creativity with exploratory testing
• Realize people dependent tests, like usability
Pipeline Integrated
Pipeline Integrated
Pipeline Patterns & AntiPatterns
Insufficient Parallelization
Ideal Time < 10-15 Mins
No more than 1 hour
Hint: Prefer a paralellizable unit testing framework. TestNG for Java and MbUnit for .Net
might be good choices.
Pipeline Patterns & AntiPatterns
Inflexible Workflow
References
http://www.romaniatesting.ro/to-automate-or-not-to-automate-testing
http://www.softwarequalitymethods.com/papers/star99%20model%20paper.pdf
http://www.continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/
http://www.testng.org/doc/documentation-main.html#parallel-tests
Referencing Lisa Crispin, Agile Testing Book 2009
Continuous Delivery Workshop Material created by Jez Humble, Martin Fowler, Tom Sulston, & Neal Ford
