Test Automation Strategies for the Agile World

TR PM Tutorial

10/14/2014 1:00:00 PM

"Test Automation Strategies for the

Agile World"

Presented by:

Bob Galen

RGalen Counsulting Group, LLC

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Bob Galen

Velocity Partners An agile methodologist, practitioner, and coach based in Cary, NC, Bob Galen helps guide companies in their adoption of Scrum and other agile methodologies and practices. Bob is a principal agile evangelist at Velocity Partners, a leading agile nearshore development partner; president of RGCG; and frequent speaker on software development, project management, software testing, and team leadership at conferences and professional groups. He is a Certified Scrum Coach, Certified Scrum Product Owner, and an active member of the Agile and Scrum Alliances. In 2013 Bob published Scrum Product Ownership–Balancing Value from the Inside Out. Reach him at [email protected].

1

Test Automation Strategies for the

Agile World

Bob GalenPresident & Principal Consultant

RGCG, LLC

[email protected]

Copyright © 2014 RGCG, LLC 2

Introduction

Bob Galen� Independent Agile Coach (CSC) at RGCG, LLC

� Principle Agile Evangelist at Velocity Partners

� Somewhere ‘north’ of 30 years overall experience ☺� Wide variety of technical stacks and business domains

� Developer first, then Project Management / Leadership, then Testing

� Senior/Executive software development leadership for 20 years

� Practicing formal agility since 2000

� XP, Lean, Scrum, and Kanban experience

� From Cary, North Carolina

� Connect w/ me via LinkedIn and Twitter @bobgalen

Bias Disclaimer:

Agile is THE BEST Methodology

for Software Development�

However, NOT a Silver Bullet!

2


Outline

� Traditional Automation – Business Case & ROI

� 3-Pillars

� Agile Test Automation Pyramid

� Agile Automation – Business Case & ROI

� Implementation Strategy

� Communication

� Wrap-up


3

Let’s start with…

Traditional Automation Strategy

� What are your current strategies towards:

� Test Automation

� Frameworks

� Tooling

� Maintenance

� ROI

� Team structure

� Get together in “pairs” and chat about this for 20

minutes. List your approaches and tactics.

� Then leverage SWOT analysis to capture your current

position. Then we’ll gather your results@


Let’s start with…

Traditional Automation Strategy

� SWOT – Often used to analyze strategic direction or

positions. List the “position”, then identify aspects of:

� Strengths

� Weaknesses

� Opportunities

� Threats

Often in Quadrants

� Force Field Analysis – again, list a position

� Forces (FOR) �

� Forces (AGAINST)


4

Traditional Automation

Business Case & ROI


It’s simple really…

� Manual testing is:

� Slow

� Labor intensive

� Not intellectually stimulating

� Error prone

� Iterative

� Did I say slow?

� And we’re testing software@right?

� So we ought to be able to automate the testing.


5

It’s simple really…

� Automated tests are:

� Fast

� Low/no errors

� Free after initial development

� Run continuously

� Did I say fast?

� Did I say ‘free’?

� And we’re testing software@right?

� So we ought to be able to automate ALL of the testing.


10Copyright © 2014 RGCG, LLC 10

� Capture size of the team

� Cost per tester – usually hourly

� # of Test cases to be run

� Time to run them

� Cyclical nature of the testing cycles (regression, integration,

system, etc.)

� Configurations

� Number of tests run per release

� Number of releases per year

� Do the math to come up with iterative testing run-time

investments@and costs

Test Automation

ROI Sample Points – Manual Testing

6


� Then contrast that against the cost for automation:

� Time / cost to establish automation infrastructure

� Time / cost to establish automated tests

� Time / cost for ongoing maintenance and results analysis

� Cost for automation tooling – depends on ‘type’, ex: Keyword

Driven

� Cost per automation developer vs. manual tester

� Automation replacement target for manual test cases

� Do the simple, discrete math to come up with cost & time

to replace manual testing run-time and ROI savings

Test Automation

ROI Sample Points – Automation Development


� Run more tests, faster and with fewer people (lower

costs)

� VALUE being driven by: Running tests!

� http://www.aspiresys.com/testautomationroi/index.php?action=__

reset

� Did I say with fewer people (testers)?

� We can hire more developers with the cost savings! And get

MORE done@

� We can even reduce the size of the bathrooms ☺

Traditional

Business Case

7


http://www.aspiresys.com/testautomationroi/index.php#ROI


http://www.aspiresys.com/testautomationroi/index.php#ROI

8

But often there is underestimation of risks…

So the ROI was always a ‘fuzzy’ estimate

� Underestimating the nature of a software project

� Architecture & design, complexity, risks, unknowns, ambiguity,

etc.

� Underestimating parallel software projects (product +

automation) complexity

� Integration, dependencies, coordination

� Underestimating ongoing support & maintenance

� Repairs & updates

� Designing and adding new tests

� Impact of new product technologies

� Underestimating skill levels for automation development


It’s also very dangerous!

Will Robinson…

� It’s commoditizing your testing activity.

Could you imagine –� Measuring developer value by LOC@and doing ROI?

� Measuring architectural integrity by complexity analysis@and doing

ROI?

� Measuring product owner value by number of backlog elements@and

doing ROI?

Based on some sort of automation introduction? Of

course not. It’s insane.

� Then why do we do it with tests@and testers?


9


� We’re all using them, so they must be good and the

approaches must be ‘correct’@right?

� However, there are challenges—

� Training

� Size

� UI-centric

� COST

� Can’t test everything (Back-end database code)

� One-size fits all strategy

� Limited developer / whole team usage

Traditional Test Automation Tools

Just a quick stop…

Typical Deployment

Level of effort


Initial design

& Architecture

Initial Test

Case Automation

Technology Disruption

Ongoing Maintenance

MORE

Typical Time & Budget

Investment

LESS

�Level of Effort -

Investment

10

Now it may sound…

� Like I’m opposed to test automation.

Truly, I’m not. I think automation

rocks! Or like I’m opposed to gaining

a ‘return’, absolutely not.

� Automation is clearly a foundation

element for increased quality and going

faster – releasing more often

� But, this traditional model or

approach of focusing on ROI@is

sort of insane


3-Pillars of Agile

Quality & Testing


11


3-Pillars

Genesis

� I’ve learned that “Balance” is important

� A sad tale of:

� Thousands of ATDD testing; Gherkin run amok

� All of them are working; continuously testing; increasing

“coverage’ and life is Good!

� BUT

� These same teams couldn’t write a cohesive User Story to save

their life

� So, where were the Acceptance Tests coming from?

3-Pillars of Agile Quality

Copyright © 2014 RGCG, LLC

22

Development & Test

Automation

• Pyramid-based Strategy:

(Unit + Cucumber +

Selenium)

• Continuous Integration

• Attack technical

infrastructure in the Backlog

• Visual Feedback –

Dashboards

• Actively practice ATDD and

BDD

Software Testing

• Risk-based testing:

Functional & Non-Functional

• Test planning @ Release &

Sprint levels

• Exploratory Testing

• Standards – checklists,

templates, repositories

• Balance across manual,

exploratory & automation

Cross-Functional Team

Practices

• Team-based Pairing

• Stop-the-Line Mindset

• Code Reviews & Standards

• Active Done-Ness

• Aggressive Refactoring of

Technical Debt

• User Stories, “3 Amigo”

based Conversations

• Whole Team Ownership of “Quality”

• Knowing the Right Thing to Build; And Building it Right

• Healthy – Agile Centric Metrics

• Steering via: Center of Excellence or Community of Practice

• Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement

12

Foundation of the 3-Pillars


23

• Whole Team Ownership of

“Quality”

• Knowing the “Right” thing to

Build AND Building it “Right”

• Healthy – Agile Centric Metrics

• Steering Required – CoE or CoP

• Strategic balance across 3

Pillars; Assessment,

Recalibration, and Continuous

Improvement

• Whole team view includes building it right,

everyone tests, everyone demo’s, etc.

• Focus on features/stories, confirmation,

conversation, and getting them staged

properly OVER testing

• 4-tier metrics: Quality, Value, Prediction, Team

• Agile strategies need light-handed “steering”;

establish a CoE (heavier weight) or a CoP

(lightweight)

• Consider finding an assessment framework

and then tying it to your strategy

measurement, recalibration, and continuous

improvement.

• Make the foundation visible thru information

radiators and metrics



24

Development &

Test Automation

• Pyramid-based

Strategy: (Unit +

Cucumber + Selenium)

• Continuous Integration

• Attack technical

infrastructure in the

Backlog

• Visual Feedback –

Dashboards

• Actively practice ATDD

and BDD

A central part of agile adoption is focusing on CI, 3-

tiered Automation development, and Dashboards to

begin incrementally building coverage for faster

feedback on changes.

100% automation is NOT the Goal!

In the interim, Hardening or Stabilization Sprints and

having a risk-based Release Train concept help

It’s important that Test or QA not ‘own’ the tooling or

all of the automation efforts. The strategy can come

from QA, but the tactical automation development is

best left to the team.

Mature teams invest in Automation, Tooling, and

Technical Debt reduction as part of Done-ness and

continually add it to their backlogs

13



25

Software Testing

• Risk-based testing:

Functional & Non-

Functional

• Test planning @

Release & Sprint levels

• Exploratory Testing

• Standards – checklists,

templates, repositories

• Balance across

manual, exploratory &

automation

Exploratory Testing (SBET with pairing) can be an

incredibly effective way to establish a whole-team,

collaborative view towards quality and testing. It also

emerges new tests.

Leverage ‘plans’ as a whole-team collaboration-

conversation mechanism; at Sprint and Release

levels.

Do not measure testing or tester progress; instead,

measure throughput, output, sprint outcomes, and

done-ness escapes at a team level.

You need a balanced test team; not everyone needs

to be able to program. But everyone needs to be

passionately skilled testers with curiosity.

Agile testing is a Risk-Based play in every Sprint and

across a release sequence.



26

Cross-Functional

Team Practices

• Team-based Pairing

• Stop-the-Line Mindset

• Code Reviews &

Standards

• Active Done-Ness

• Aggressive Refactoring

of Technical Debt

• User Stories – 3 Amigo

based Conversations

One of the hardest areas to get ‘right’ culturally. It

needs leadership alignment from Quality/Testing to

Product to Development and a consistent voice of

whole-team approaches.

This is where LEAN Thinking lives, where whole-

team collaboration happens, where professionalism

and craftsmanship are held dear.

I like the view of testers becoming the VOC,

champions of quality, and consistent questioners of

what is being build. Are we solving the right

problems@as simply as possible. Notions of Minimal

Viable Product / Feature help with focus.

And yes Virginia, there ARE standards, templates,

and a focus on x-team consistency!

14

Agile Test Automation

Pyramid




Pyramid

� Coined by Mike Cohn, ~ 2005

� Establishes the validity of turning Traditional Automation

– upside down

� Invests:

� Most effort (%) in Unit Tests

� Moderate effort (%) in Middle-tier tests (business logic, API,

component, feature)

� Least effort (%) in UI-centric, top down tests

� Granularity of the tests is part of the strategy

15



Pyramid

Often:

� Tied to Definition-of-Done from a maintenance

perspective

� You break it, you fix it

� Percentages vary; intent does not

� Whole-team ownership of automation

� Although it skews at either end of the pyramid

� Run as much as possible

� Check-in, Overnight, Cyclically

� Prime Directive = Real-time Feedback


Pyramid


+80%, UI Centric Automation

0-10%, Service or Middle Tier Automation

0-10%, Unit level Automation

0-10%, UI Centric Automation

20-40%, Service or Middle Tier Automation

50-60%, Unit level Automation

Traditional

Agile

16

Agile Test Automation PyramidMike Cohn; Lisa Crispin & Janet Gregory

http://behaviordrivendevelopment.wikispaces.com/Testing


Brainstorm…

Agile, Multi-tiered Automation

� Get together in small groups of 4-6 to discuss

� Take a few minutes and think about your current

automation approaches:

� Tooling, approaches & strategies, strengths, weaknesses,

opportunities, maintenance challenges, future technology, etc.

� What sorts of adjustments would you need to make to

take this approach?

� What would be the largest challenges in taking this

approach? How might you overcome them?

� Do you “buy” the whole-team view to automation?


17


ROI


What agile has exposed to me…

wrt/Automation & Testing

� Testers vs. a Whole Team view� Build teams in ratios – developers vs. testers

� Instead invest in cross-functional teams composed of the skills to design,

construct, test, and deliver excellent products

� Only testers write test automation

� The entire team writes test automation and, oh by the way, can test as

well

� Only testers test the application

� The whole team participates in writing / running automation, functional

testing, exploratory testing, regression testing, etc.


18



� Traditional vs. Open Source

Tools� Instead of leveraging singular, UI-centric tools

� Leverage multiple tools at all-tiers of your application stack

� Higher cost

� Lower cost and a sense of “Community”

� Developers won’t use them; costs

� Everyone is leveraging the same tool-sets; AND integrating them in

Continuous Integration and Continuous Deployment




� People are your primary value

proposition� Instead of looking at testers as a commodity

� Consider good testers to be as valuable as any other team member

� Instead of thinking that testing is a by-rote exercise with little intellectual

pursuit

� Testing is a profession. Testers are valuable craftspeople. Testing is an

intellectual exercise.

� Instead of thinking the value is in the test cases

� The value is in the testers (people) continuously reevaluating the right

things to test at the right time@for valuable feedback


19


http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html

The NEW Test Automation

Business Case

� So, don’t

� Count pennies

� Count heads

� Count cycles

� Work as part of a TEAM

� Focus in on testing continuously – just the right things at just the

right time

� Providing feedback, transparency, and reaction

� Allowing your teams time to learn and improve

� Solving your customers’ problems; delivering value


20


� Run more tests, faster, unattended, continuous

feedback, learning & adjustments

� Yes, saving time. Now what to do with it???

� Drive your new VALUE by:

� Early feedback for adjustments—risk avoidance, learning, solving

real problems

� Having more time to continuously refine your test cases

(automated and not)

� Having more time to focus on building Quality into your products

� Creatively testing the application (Exploratory Testing, Test Idea

Generation, Test Design)

The NEW Test Automation

Business Case

So, with the additional time…

INVEST in your testers & quality

� In building In Quality

� In attacking the customers REAL problems

� In better testing

� In more balanced automation

� In team training; in slack time

� In better collaboration

� In doing more than thought possible

� In creative solutions

� In adapting to new technologies

� In having FUN


21

How do we “Sell”

this change?

� You don’t sell it from a testing

perspective. Nor from a

bean-counting perspective.

� You allow the results to begin speaking for themselves

as we begin focusing on and talking about—

� How one builds truly innovative and creative software teams

� That understand and solve customer problems

� With simple solutions that fundamentally work

� And do so iteratively & quickly, while nimbly adjusting to

customer feedback


But Bob…

� That doesn't work in the “Real World”!

� My boss would never go for it without ROI justification

and hard measures

� Testing IS a commodity

� We’ve done automation perfectly – seeing none of the

issues/risks you’ve pointed out


22

But Bob…

� Think about what I’m saying; taking

the time and@

� Invest in people

� Invest in better quality & testing

� Invest in better product solutions to customer problems

� Invest in improved workflow and delivering high customer value

WOW – how crazy is that?

� Meet in small groups of 2-3 (pro / con) and chat for 10

minutes about the “sides” and the possibility for change?

� Then we’ll debrief@



Implementation Strategy


Automation is one of the foundational investments of effective Agile

Adoptions. It’s not a choice, it’s an imperative! It’s a cost of doing business.

23

Implementation Strategy

Typically a 5-Phased Strategy

1. Strategy – This session: 3 Pillars + Pyramid + Team

2. Team Formation

3. Tools Selection

4. Training & Infrastructure Development

5. Automation Development

6. Care & Feeding


Phase 2

Team Formation

� GOAL@whole team ownership

� Skill-set discussion; SDET vs. Test Professional?

� May influence

� Development focus on Unit Testing

� Automation team to “gain momentum”

� Agile execution (Scrum, Kanban, transparency, demo, etc)

� Connect the dots architecturally

� Bias: I like to lead automation with a Test Automation

Architect

� Rather than with someone from the development team


24

Phase 3

Tools Selection

� Open Source vs. Commercial

� Singular, all-purpose vs. Tool-kit

� Cobble together or Integrate disparate tools

� Always perform a “Bake off” within your teams

� If Open Source, consider support implications

� Connect to development tooling

� Including CI and CD

� Additional Considerations: Test data, virtualization,

DevOps

� Be prepared to iterate, learn, and possibly fail


Phase 4

Training & Infrastructure

� Design, coding, and documentation standards

� Templates

� Community of Practice

� Example code, standards, forum, collaborative groups

� Models:

� Keyword, Data-driven, Table-driven, DSL, Model-driven, others

� Test Suite; Hierarchy, Naming

� Commercial: invest in training and/or consulting

� Open Source: hire and/or acquire direct expertise

� Also can build the expertise


25

Phase 5

Automation Development

� Move it to your teams ASAP

� Make it a part of Definition-of-Done

� It should become a natural outcome and not a choice or question

� Complete it within each sprint

� SAFe as a goal model for completing work in the same sprint!

� Salesforce as well

� It needs to be a negotiated part of your Product Backlog

� Have % targets;

� For every story:

� The TEAM decides the requisite automated tests at each level of

the pyramid?


Phase 6

Care & Feeding

� Automation evolution roadmap

� Merge with your architectural look-ahead

� Merge with your Product Roadmap

� Factor in ongoing automation investments

� Refactoring & Repairs as part of your Technical Debt

handling

� Consider it at the same level of investment as your

application code

� Stop-the-Line

� Commitment


26

Communication

� Leadership Vision

� High-level Plans, Strategic sharing with your teams

� Leadership collaboration (Development + QA)

� Intent to doggedly pursue automation

� Information Radiators

� Roadmaps & Product Backlogs

� Sprint & Release Reviews

� Even though it’s hard to show “working software”

� Let the “process” transparently communicate your progress,

impediments, plans, challenges


Let’s end with…

Agile Automation Strategy

� Based on what you’ve heard, discussed, learned@

� Get together in “pairs” and chat about this for 20

minutes. List your strategic approaches. Then leverage

FFA to capture your driving forces FOR and AGAINST

moving towards Agile Automation

� Most importantly, focus on strategies to overcome your

obstacles.

� Then we’ll gather your results@


27



Wrap-up

THANK YOU!

� I hope I’ve added to your previous

assumptions and understanding

� Inspired you to change your view towards Automation

ROI and investment

� Introduced you to models for

attacking Agile Automation Strategy

and ROI

� Final questions or discussion?

5454

28

Contact Info

Bob GalenPrincipal Consultant,

RGalen Consulting Group, L.L.C.

Experience-driven agile focused training, coaching & consulting

Cell: (919) 272-0719

[email protected] www.rgalen.com

[email protected] www.velocitypartners.net

BlogsProject Times - http://www.projecttimes.com/robert-galen/

BA Times - http://www.batimes.com/robert-galen/

Podcast on all things ‘agile’ - http://www.meta-cast.com/


References


29

Links

� http://www.agileengineeringdesign.com/2012/01/7-

deadly-sins-of-automated-software-testing/

� Link to PDF of Pillar-1 chapter

� Links to “old” STP 3-part automation series

� Link to Mary’s – Agile Test Manager article


Open Source Tools

� Unit Level

� x-Unit

� Junit, nunit, phpunit,

� ATDD

� FitNesse, Robot Framework

� BDD

� Cucumber, JBehave

� UI

� Selenium, Watir, HP-UTF


30

…DD

much from the XP space

� TDD - Test Driven Development

� Test-first; Unit Tests as a design aid

� Debate around creating the unit tests “first”

� ATDD - Acceptance Test Driven Development

� User Story – Acceptance Tests

� Executable Requirements

� BDD - Behavior Driven Development

� jBehave, Cucumber; Gherkin – Given, When, Then

� More useful phrasing

� Tests as functional documentation

� Product Owners, Customer – writing tests


Agile Testing QuadrantsBrian Marick; Lisa Crispin & Janet Gregory


Exploratory testing

Scenarios

Usability testing

UAT

Alpha / Beta

Unit tests

Component tests

API tests

Functional tests

Story tests

Examples

Prototypes

Simulations

Performance testing

Load testing

Security testing

Non-functional requirements

Q1

Q2 Q3

Q4

Automated &

Manual

Automated &

Manual

Manual

Automation,

Tools, and

Manual

Business Facing

Technology Facing

Supporting the Team

Critiq

ue the Product