Making Test Automation Work in Agile Projects

StarWest 2009

Lisa Crispin With Material from Janet Gregory

Topics – Making Automation Work

Learn ways to:   Overcome common obstacles   Use automation to “keep up”   Succeed with the “whole team” approach   Use automation to enhance ET   Evaluate and implement tools   Use ROI to help succeed

Why Automate?

  Free up time for most important work   Repeatable   Safety net   Quick feedback   Help drive coding   Manual tests are error prone   Tests provide documentation

  You know what your system does

Barriers to Test Automation

What’s holding you back?

Fear Changing code

Skills

Not designed For testability

Pain and Fear  Programmers don’t feel manual test pain  Testers treated as safety net  Fear

  Programmers lack testing skills   Testers lack programming skills

It’s Worth It  ROI – explain to management   “Present value” of automated tests  Acknowledge hump of pain

Initial Investment

  Hump of pain   Legacy code, changing code   Tools, infrastructure, time

What Affects ROI

  Good test practices increase ROI   Simple, well-designed, refactored tests   Test resources improve over time

  Poor test practices reduce ROI   Tests are hard to understand   Tests are hard to maintain

Questions on Why Automate?

Small Group Exercise

 Write obstacles hindering your team on small Post-Its..

 Stick them on the wall or flip chart.  Group similar ones. What patterns emerge?  Are there any unexpected barriers?  Share patterns, surprising outcomes with the

class.

Getting Over the Hump

 The test automation pyramid  The agile testing quadrants  What should be automated  What shouldn't  Difficult areas

Test Automation Pyramid

Agile Testing Quadrants

What Should We Automate   Quadrant 1 Tests

  Unit, Component, TDD (all)   Quadrant 2 Tests

  Behind GUI, API, web services, GUI   Quadrant 3 Tests

  Leverage automation where useful   Setting up tests, tools, data generation

  Quadrant 4 Tests   Load, performance, stress

What Else Can We Automate?

Automate tedious or repetitive tasks

  Testing-related or otherwise   Continuous integration, builds   Deployments   Checking for updates   Parsing, comparing output   Automating tasks for business

What We Shouldn’t Automate

  Wizard of Oz tests   Paper prototyping   Usability tests / look and feel   Customer adhoc testing   Exploratory testing   Tests that never fail ???   One-off tests   Tests covered elsewhere

Where Should We Be Careful   GUI tests

  Need to test the GUI   Can do ATDD - Watch ROI

  Automating end to end tests   Don't go overboard, automating every path   Aim for minimum feature coverage

Remember the pyramid   Push testing down as low as you can   Robust lower-level tests = better ROI

Hard to Automate? Legacy code   Hard to automate, or just lack of skill?   “Working Effectively with Legacy Code” –

Feathers   “Strangling” – Fowler, Thomas

  http://martinfowler.com/bliki/StranglerApplication   Mike Thomas article: http://samoht.com/wiki/wiki.pl?

Software_Development

Discussion

Where should you start? Or do next?   Is there low-hanging fruit?   What’s the best ROI?

Agile Automation Strategy

 What hurts the most   Layered approach  Applying agile principles  Choosing the right tools

What Hurts the Most

 Keep an impediment backlog  What’s biggest obstacle?

  Time   Tools   Code design   Testability

Multi-Layered Approach Example:   TDD delivers unit, component tests   ATDD delivers API level, GUI tests   GUI smoke tests as regression safety net For new or legacy code

Specialists Needed?

•  May bring in shared or outside resources   Performance, load   Security   User experience design   Toolsmiths

  Dev team still responsible for completing all tests

  Multiple viewpoints provide synergy

Discussion

  What skills would help your team overcome automation barriers?

  What skills are missing from your team? Where could you get them?

Applying Agile Principles

  Whole team approach   Learn by doing   Simplicity   Iterative feedback   Small chunks

Whole-Team Approach

  Team responsible for testing, quality   Team responsible for testing activities

  Includes automation   Whole team has all the skills needed   Team designs for ease of test automation

Learn by Doing

  Courage – don’t be afraid to fail   Use agile coding practices for automation

  Object-oriented, libraries, modules   Simple design   Pairing   Refactoring   Test-first if scripts have logic

Simplicity

  Address one or two needs at a time   Understand the problem first   Try simplest approach first   Work in small chunks, thin slices   Incremental & iterative

Iterative Feedback

  Commit – Try a new tool/framework for “n” iterations

  Plan automation tasks for each iteration   Use retrospective to evaluate

Automate a Steel Thread at a Time: Example

4-step UI to upload item information from csv file into inventory system

  Thread 1: All four pages, user can navigate   Thread 2: Select age category, enter info on page 1,

display on page 2, browse and upload file on page 2   Thread 3: Validate data in file, display on page 3   Thread 4: Persist data, display ‘success’ message on

Steel Thread Example A steel thread diagram, with “passes” numbered 1, 2. 3 & 4.

Small Group Exercise You’re a development team at Agile Testing

Toys. The story on the next slide is your next priority.

Break the story down into thin slices or threads whose tests can be automated. Prioritize them. How could the functionality be delivered incrementally?

Story …

Assumptions/ Conditions of Satisfaction   User has already entered shipping address.   User can choose different options for different items.   The options are Ground, 2 day and Overnight.   Items > 20 lbs are Ground only   API to cost calculator is available, takes shipping type,

postal code and weight, returns shipping cost.

As an Agile Testing Toys shopper, I want to select a shipping type for my items during checkout and see the shipping cost.

Questions on

Creating an Automation Strategy?

Choosing Your Tools

One size does not fit all !

Find Time for Evaluating 1.  Get commitment for time to research 2.  Budget time

  Individual or group 3.  Determine your requirements 4.  Do some basic research 5.  Compile a list of tools to consider 6.  Try the ones which meet your criteria

Understand the Purpose

•  What’s being automated?   e.g. Ajax, SSL support; load; embedded s/w   Speed up exploratory testing, build test data

•  Existing tools, environment   e.g.. integration with build process   Reporting needs

•  Who’s writing, maintaining the tests? •  Who’s using the tests? What for? •  What are your constraints?

What Fits Your Situation

  Existing skills on team   Language of application under test   Collaboration needs   Utilities (automating tedious tasks) vs.

testing   Life span, future use of tests

Vendor Tools - Pros

•  Existing expertise •  Some built on open-source libraries •  Fast ramp-up for non-programmers •  Perceived as safe choice •  Training, support •  Part of existing tool set •  May have robust features

Vendor Tools - Cons

•  Tend to be heavyweight •  Tend to be programmer-unfriendly •  Scripts may be brittle, high-maintenance

•  Capture-playback problematic •  Not designed for long-term maintainability

•  Can be pricey

Open-Source Tools - Pros

•  Designed by test-infected programmers •  Designed for agile environments •  Designed for maintainability •  Programmer-friendly •  May have excellent support, tutorials, doc •  Easily customized •  Low up-front cost

Open-Source Tools - Cons

•  May be difficult for non-programmers •  Depends on the tool/framework

•  Future enhancements may be uncertain •  Training, support can be an issue •  Be sure to look for active development

community

Home-Brewed - Pros

•  Programmer-friendly •  Integration with app, IDEs

•  Development framework may support •  Rails, Ruby, Groovy

•  Can build on top of existing framework •  Fit, Slim, Watir, RSpec

•  Specifically tailored to needs •  Someone’s there to address problems

Home-Brewed - Cons

•  Team needs enough bandwidth, expertise •  Reporting framework •  Allow test specification by non-

programmers •  Could be harder sell to management

Example: A Java Team’s Choices

  IntelliJ Idea, Eclipse for IDEs   CruiseControl, Hudson for CI   JUnit for TDD, unit, component   FitNesse for functional, behind GUI   Canoo WebTest for GUI regression smoke tests   Watir to aid exploratory testing   JMeter for load, performance testing   Perl scripts for comparing files, making files

human-readable   Java programs for concatenating forms, cover

letters

Example: UPM Team’s Choices

  Visual Studio, Resharper   MBUnit   Hudson for CI   FitNesse/SWAT, DBFit for GUI, functional tests   ncover for code coverage   Confluence wiki for documentation

Discussion

Any tool selection success stories to share?

Questions about choosing tools?

Making Test Automation Work

  Time to learn, implement, design   Testable architecture   Test data   Managing tests

Time To Do It Right

  Limit scope   Write automation task cards   Quality must be goal   Long-term, will let you go faster

Testable Architecture   Layered architecture

  e.g.. UI, business logic, data access   Ports and Adapters pattern

  App can work without UI or database   Ports accept outside events   Adapters convert for human or automated

Test Data

  Avoid database access when possible   Setup/Teardown

  Independent, rerunnable tests   Canonical data

  Refresh before each test run   Customizable data for ET   Production-like data

  Get customers to provide example data

Managing Automated Tests

  Documenting tests   Tests as documentation   Finding what you need

  Reporting results   Analyze failures   Let CI do the work

  Test coverage

Example: My Team’s Tool Choices

Test Management Tools   Tool may include management framework

  FitNesse   Wiki – documentation + executable tests   Hierarchy

  Rasta – spreadsheets to organize tests   Twist – uses Eclipse IDE

  What problem are you trying to solve?   Simplest approach   Use SCCS for tests

Small Group Exercise   Look at the barriers from the first exercise.   What’s one thing you can do when you get

back to the office to overcome one of your team’s barriers? Write it on a post-it and put it next to the barrier on the flip chart paper.

  Share your first action item with the class

Make Automation Work for You   Don’t try to find one perfect solution   Start simple   Budget time to evaluate, implement   Use feedback, retrospectives   Apply agile principles, practices   Budget time to refactor, try new alternatives

Remember

  It’s a team problem!   Build foundation of core agile practices   Experiment   Baby steps

Questions? “Aha” Moments?

Some Agile Testing Tool Resources •  awta.wikispaces.com/2009ToolsList •  softwareqatest.com/qattls1.html •  opensourcetesting.org •  nunit.org/index.php •  webtest.canoo.com •  testingfaqs.org •  junit.org •  wtr.rubyforge.org •  fit.c2.com •  gojko.net/fitnesse/dbfit •  fitnesse.org •  selenium.openqa.org •  jameleon.sourceforge.net •  artofunittesting.com/Chapters/Tools_and_frameworks

Now Available

Agile Testing: A Practical Guide for Testers and Agile Teams

By Lisa Crispin and Janet Gregory www.agiletester.ca

Test Patterns

Xunit Test Patterns: Refactoring Test Code

By Gerard Meszaros

Bridging the Communication Gap Specification By Example and Acceptance Testing

Gojko Adzic

Making Test Automation Work in Agile Projects - Agile Testing...

Documents

Automation Functional Testing in Agile Projects

SPLs Agile development & Industry automation

Estimation Agile Projects

Building QA Automation Using Agile - Synerzip€¦ · Introduction • Topic: Building QA Automation Using Agile Automated testing is a key requirement for effective Agile development

Test automation in an agile environment

Planning Agile Projects

Agile Test Automation - WordPress.com · Agile Test Automation Key Requirements for Functional and Performance Testing Introduction Agile test automation may sound like a contradiction

Making Test Automation Work in Agile Projectslisacrispin.com/downloads/AgileTDAutomation.pdf · 1 Making Test Automation Work in Agile Projects Agile Testing Days 2010 Lisa Crispin

Test Automation in Agile

Agile Td Automation

Automation testing in Agile project

Testing automation in agile environment

Integrate Test Activities in Agile Projects integrate test... · Integrate Test Activities in Agile Projects 2 PAGE • The automation of testing tasks (e.g. test specification and/or

Agile Automation Strategies and Frameworks

Test Automation: Agile Enablement for Data Warehousing and ...files.meetup.com/2625872/Test automation... · Test Automation: Agile Enablement for Data Warehousing and Business Intelligence

Making Test Automation Work in Agile Projects · 2012-12-29 · 1 Making Test Automation Work in Agile Projects StarWest 2009 Lisa Crispin With Material from Janet Gregory

Agile Automation - National Multifamily Housing Council...@ApartmentWire #OPTECH18 Agile Automation. How to Make Marketing Automation Thrive . in a High-Touch, Customer-Centric Industry

AGILE AND AUTOMATION CONCLAVE 2018 · AGILE AND AUTOMATION CONCLAVE 2018 AI in Agile Workshop Anubhav F Gupta Raghavendra Meharwade. Agile and Automation Conclave 2018 ... • Lead

Making Test Automation Work in Agile Projects 2012

Strategies for agile software test automation