Making Test Automation Work in Agile Projects - Agile Testing …lisacrispin.com/downloads/StarWestAgileAutomation.pdf · 2012-12-29 · Making Test Automation Work in Agile Projects

1

Making Test Automation Work in Agile Projects

StarWest 2009

Lisa Crispin With Material from Janet Gregory

Copyright 2009 Lisa Crispin and Janet Gregory

2

Topics – Making Automation Work

Learn ways to:   Overcome common obstacles   Use automation to “keep up”   Succeed with the “whole team” approach   Use automation to enhance ET   Evaluate and implement tools   Use ROI to help succeed


3

Why Automate?

  Free up time for most important work   Repeatable   Safety net   Quick feedback   Help drive coding   Manual tests are error prone   Tests provide documentation

  You know what your system does


4

Barriers to Test Automation

What’s holding you back?

Fear Changing code

Skills

?????

Not designed For testability


5

Pain and Fear  Programmers don’t feel manual test pain  Testers treated as safety net  Fear

  Programmers lack testing skills   Testers lack programming skills


6

It’s Worth It  ROI – explain to management   “Present value” of automated tests  Acknowledge hump of pain


Initial Investment

  Hump of pain   Legacy code, changing code   Tools, infrastructure, time

Time

Effo

rt

7

Copyright 2009 Lisa Crispin and Janet Gregory Copyright 2008: Janet Gregory, DragonFire Inc.

8

What Affects ROI

  Good test practices increase ROI   Simple, well-designed, refactored tests   Test resources improve over time

  Poor test practices reduce ROI   Tests are hard to understand   Tests are hard to maintain


9

Questions on Why Automate?


10

Small Group Exercise

 Write obstacles hindering your team on small Post-Its..

 Stick them on the wall or flip chart.  Group similar ones. What patterns emerge?  Are there any unexpected barriers?  Share patterns, surprising outcomes with the

class.


11

Getting Over the Hump

 The test automation pyramid  The agile testing quadrants  What should be automated  What shouldn't  Difficult areas


12

Test Automation Pyramid


13

Agile Testing Quadrants


What Should We Automate   Quadrant 1 Tests

  Unit, Component, TDD (all)   Quadrant 2 Tests

  Behind GUI, API, web services, GUI   Quadrant 3 Tests

  Leverage automation where useful   Setting up tests, tools, data generation

  Quadrant 4 Tests   Load, performance, stress

14


15

What Else Can We Automate?

Automate tedious or repetitive tasks

  Testing-related or otherwise   Continuous integration, builds   Deployments   Checking for updates   Parsing, comparing output   Automating tasks for business


What We Shouldn’t Automate

  Wizard of Oz tests   Paper prototyping   Usability tests / look and feel   Customer adhoc testing   Exploratory testing   Tests that never fail ???   One-off tests   Tests covered elsewhere

16


17

Where Should We Be Careful   GUI tests

  Need to test the GUI   Can do ATDD - Watch ROI

  Automating end to end tests   Don't go overboard, automating every path   Aim for minimum feature coverage

Remember the pyramid   Push testing down as low as you can   Robust lower-level tests = better ROI


18

Hard to Automate? Legacy code   Hard to automate, or just lack of skill?   “Working Effectively with Legacy Code” –

Feathers   “Strangling” – Fowler, Thomas

  http://martinfowler.com/bliki/StranglerApplication   Mike Thomas article: http://samoht.com/wiki/wiki.pl?

Software_Development


19

Discussion

Where should you start? Or do next?   Is there low-hanging fruit?   What’s the best ROI?


20

Agile Automation Strategy

 What hurts the most   Layered approach  Applying agile principles  Choosing the right tools


21

What Hurts the Most

 Keep an impediment backlog  What’s biggest obstacle?

  Time   Tools   Code design   Testability


22

Multi-Layered Approach Example:   TDD delivers unit, component tests   ATDD delivers API level, GUI tests   GUI smoke tests as regression safety net For new or legacy code


23

Specialists Needed?

•  May bring in shared or outside resources   Performance, load   Security   User experience design   Toolsmiths

  Dev team still responsible for completing all tests

  Multiple viewpoints provide synergy


24

Discussion

  What skills would help your team overcome automation barriers?

  What skills are missing from your team? Where could you get them?


Applying Agile Principles

  Whole team approach   Learn by doing   Simplicity   Iterative feedback   Small chunks


26

Whole-Team Approach

  Team responsible for testing, quality   Team responsible for testing activities

  Includes automation   Whole team has all the skills needed   Team designs for ease of test automation


27

Learn by Doing

  Courage – don’t be afraid to fail   Use agile coding practices for automation

  Object-oriented, libraries, modules   Simple design   Pairing   Refactoring   Test-first if scripts have logic


28

Simplicity

  Address one or two needs at a time   Understand the problem first   Try simplest approach first   Work in small chunks, thin slices   Incremental & iterative


29

Iterative Feedback

  Commit – Try a new tool/framework for “n” iterations

  Plan automation tasks for each iteration   Use retrospective to evaluate


30

Automate a Steel Thread at a Time: Example

4-step UI to upload item information from csv file into inventory system

  Thread 1: All four pages, user can navigate   Thread 2: Select age category, enter info on page 1,

display on page 2, browse and upload file on page 2   Thread 3: Validate data in file, display on page 3   Thread 4: Persist data, display ‘success’ message on

page 4


Steel Thread Example A steel thread diagram, with “passes” numbered 1, 2. 3 & 4.

31


32

Small Group Exercise You’re a development team at Agile Testing

Toys. The story on the next slide is your next priority.

Break the story down into thin slices or threads whose tests can be automated. Prioritize them. How could the functionality be delivered incrementally?


33

Story …

Assumptions/ Conditions of Satisfaction   User has already entered shipping address.   User can choose different options for different items.   The options are Ground, 2 day and Overnight.   Items > 20 lbs are Ground only   API to cost calculator is available, takes shipping type,

postal code and weight, returns shipping cost.

As an Agile Testing Toys shopper, I want to select a shipping type for my items during checkout and see the shipping cost.


34

Questions on

Creating an Automation Strategy?


Choosing Your Tools

One size does not fit all !


36

Find Time for Evaluating 1.  Get commitment for time to research 2.  Budget time

  Individual or group 3.  Determine your requirements 4.  Do some basic research 5.  Compile a list of tools to consider 6.  Try the ones which meet your criteria


37

Understand the Purpose

•  What’s being automated?   e.g. Ajax, SSL support; load; embedded s/w   Speed up exploratory testing, build test data

•  Existing tools, environment   e.g.. integration with build process   Reporting needs

•  Who’s writing, maintaining the tests? •  Who’s using the tests? What for? •  What are your constraints?


38

What Fits Your Situation

  Existing skills on team   Language of application under test   Collaboration needs   Utilities (automating tedious tasks) vs.

testing   Life span, future use of tests


39

Vendor Tools - Pros

•  Existing expertise •  Some built on open-source libraries •  Fast ramp-up for non-programmers •  Perceived as safe choice •  Training, support •  Part of existing tool set •  May have robust features


40

Vendor Tools - Cons

•  Tend to be heavyweight •  Tend to be programmer-unfriendly •  Scripts may be brittle, high-maintenance

•  Capture-playback problematic •  Not designed for long-term maintainability

•  Can be pricey


41

Open-Source Tools - Pros

•  Designed by test-infected programmers •  Designed for agile environments •  Designed for maintainability •  Programmer-friendly •  May have excellent support, tutorials, doc •  Easily customized •  Low up-front cost


42

Open-Source Tools - Cons

•  May be difficult for non-programmers •  Depends on the tool/framework

•  Future enhancements may be uncertain •  Training, support can be an issue •  Be sure to look for active development

community


43

Home-Brewed - Pros

•  Programmer-friendly •  Integration with app, IDEs

•  Development framework may support •  Rails, Ruby, Groovy

•  Can build on top of existing framework •  Fit, Slim, Watir, RSpec

•  Specifically tailored to needs •  Someone’s there to address problems


44

Home-Brewed - Cons

•  Team needs enough bandwidth, expertise •  Reporting framework •  Allow test specification by non-

programmers •  Could be harder sell to management


45

Example: A Java Team’s Choices

  IntelliJ Idea, Eclipse for IDEs   CruiseControl, Hudson for CI   JUnit for TDD, unit, component   FitNesse for functional, behind GUI   Canoo WebTest for GUI regression smoke tests   Watir to aid exploratory testing   JMeter for load, performance testing   Perl scripts for comparing files, making files

human-readable   Java programs for concatenating forms, cover

letters


46

Example: UPM Team’s Choices

  Visual Studio, Resharper   MBUnit   Hudson for CI   FitNesse/SWAT, DBFit for GUI, functional tests   ncover for code coverage   Confluence wiki for documentation


47

Discussion

Any tool selection success stories to share?


48

Questions about choosing tools?


49

Making Test Automation Work

  Time to learn, implement, design   Testable architecture   Test data   Managing tests


50

Time To Do It Right

  Limit scope   Write automation task cards   Quality must be goal   Long-term, will let you go faster


51

Testable Architecture   Layered architecture

  e.g.. UI, business logic, data access   Ports and Adapters pattern

  App can work without UI or database   Ports accept outside events   Adapters convert for human or automated

users


52

Test Data

  Avoid database access when possible   Setup/Teardown

  Independent, rerunnable tests   Canonical data

  Refresh before each test run   Customizable data for ET   Production-like data

  Get customers to provide example data


53

Managing Automated Tests

  Documenting tests   Tests as documentation   Finding what you need

  Reporting results   Analyze failures   Let CI do the work

  Test coverage


54

Example: My Team’s Tool Choices


55

Test Management Tools   Tool may include management framework

  FitNesse   Wiki – documentation + executable tests   Hierarchy

  Rasta – spreadsheets to organize tests   Twist – uses Eclipse IDE

  What problem are you trying to solve?   Simplest approach   Use SCCS for tests


56

Small Group Exercise   Look at the barriers from the first exercise.   What’s one thing you can do when you get

back to the office to overcome one of your team’s barriers? Write it on a post-it and put it next to the barrier on the flip chart paper.

  Share your first action item with the class


57

Make Automation Work for You   Don’t try to find one perfect solution   Start simple   Budget time to evaluate, implement   Use feedback, retrospectives   Apply agile principles, practices   Budget time to refactor, try new alternatives


58

Remember

  It’s a team problem!   Build foundation of core agile practices   Experiment   Baby steps


59

Questions? “Aha” Moments?


60

Some Agile Testing Tool Resources •  awta.wikispaces.com/2009ToolsList •  softwareqatest.com/qattls1.html •  opensourcetesting.org •  nunit.org/index.php •  webtest.canoo.com •  testingfaqs.org •  junit.org •  wtr.rubyforge.org •  fit.c2.com •  gojko.net/fitnesse/dbfit •  fitnesse.org •  selenium.openqa.org •  jameleon.sourceforge.net •  artofunittesting.com/Chapters/Tools_and_frameworks

Copyright 2009 Lisa Crispin and Janet Gregory 61

Now Available

Agile Testing: A Practical Guide for Testers and Agile Teams

By Lisa Crispin and Janet Gregory www.agiletester.ca

Copyright 2009 Lisa Crispin and Janet Gregory 62

Test Patterns

Xunit Test Patterns: Refactoring Test Code

By Gerard Meszaros

Copyright 2009 Lisa Crispin and Janet Gregory Copyright 2008 Janet Gregory, DragonFire

63

Bridging the Communication Gap Specification By Example and Acceptance Testing

Gojko Adzic

Documents

Making Test Automation Work in Agile Projects - Agile Testing …lisacrispin.com/downloads/StarWestAgileAutomation.pdf · 2012-12-29 · Making Test Automation Work in Agile Projects