Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Stop the Test Automation ROI-based Justification Insanity
Bob Galen President & Principal Consultant
RGCG, LLC [email protected]
Copyright © 2012 RGCG, LLC 3
Introduction Bob Galen n Somewhere ‘north’ of 30 years experience J n Various lifecycles – Waterfall variants, RUP, Agile, Chaos… n Various domains – SaaS, Medical, Financial Services, Computer
& Storage Systems, eCommerce, and Telecommunications n Developer first, then Project Management / Leadership, then
Testing n Leveraged ‘pieces’ of Scrum in late 90’s; before ‘agile’ was ‘Agile’ n Agility @ Lucent in 2000 – 2001 using Extreme Programming n Formally using Scrum since 2000 n Currently an independent Agile Coach (CSC – Certified Scrum
Coach, one of 50 world-wide) q at RGCG, LLC and Director of Agile Solutions at Zenergy Technologies
n From Cary, North Carolina n Connect w/ me via LinkedIn and Twitter if you wish…
Bias Disclaimer: Agile is THE BEST Methodology for Software Development…
However, NOT a Silver Bullet!
Outline
n Traditional Automation ROI n Traditional Business Case n Risks & Reality n Agile as an Automation Disruptor n New Business Case n Investments & Selling n The Big But n Wrap-up
Copyright © 2012 RGCG, LLC 4 4
It’s simple really…
n Manual testing is: q Slow q Labor intensive q Not intellectually stimulating q Error prone q Iterative q Did I say slow?
n And we’re testing software…right? q So we ought to be able to automate the testing.
Copyright © 2012 RGCG, LLC 5 5
It’s simple really…
n Automated tests are: q Fast q Low/no errors q Free after initial development q Run continuously q Did I say fast? q Did I say ‘free’?
n And we’re testing software…right? q So we ought to be able to automate the testing.
Copyright © 2012 RGCG, LLC 6 6
7 Copyright © 2012 RGCG, LLC 7
ü Capture size of the team ü Cost per tester – usually hourly ü # of Test cases to be run ü Time to run them ü Cyclical nature of the testing cycles (regression, integration,
system, etc.) ü Configurations ü Number of tests run per release ü Number of releases per year
n Do the math to come up with iterative testing run-time investments…and costs
Test Automation ROI Sample Points – Manual Testing
8 Copyright © 2012 RGCG, LLC 8
n Then contrast that against the cost for automation: ü Time / cost to establish automation infrastructure ü Time / cost to establish automated tests ü Time / cost for ongoing maintenance and results analysis ü Cost for automation tooling – depends on ‘type’, ex: Keyword
Driven ü Cost per automation developer vs. manual tester ü Automation replacement target for manual test cases
n Do the simple, discrete math to come up with cost & time
to replace manual testing run-time and ROI savings
Test Automation ROI Sample Points – Automation Development
9 Copyright © 2012 RGCG, LLC 9
n Run more tests, faster and with fewer people (lower costs)
n VALUE being driven by: Running tests! q http://www.aspiresys.com/testautomationroi/index.php?
action=__reset
n Did I say with fewer people (testers)? q We can hire more developers with the cost savings! And get
MORE done… q We can even reduce the size of the bathrooms J
Traditional Business Case
Copyright © 2012 RGCG, LLC 10
http://www.aspiresys.com/testautomationroi/index.php#ROI
Copyright © 2012 RGCG, LLC 11
http://www.aspiresys.com/testautomationroi/index.php#ROI
But often there is underestimation of risks… So the ROI was always a ‘fuzzy’ estimate § Underestimating the nature of a software project
§ Architecture & design, complexity, risks, unknowns, ambiguity, etc.
§ Underestimating parallel software projects (product + automation) § Integration, dependencies, coordination
§ Underestimating ongoing support & maintenance § Repairs & updates § Designing and adding new tests § Impact of new product technologies
§ Underestimating skill levels for automation development
Copyright © 2012 RGCG, LLC 12 12
It’s also very dangerous! Will Robinson… § It’s commoditizing your testing activity. Could you imagine –
§ Measuring developer value by LOC…and doing ROI? § Measuring architectural integrity by complexity analysis…and doing
ROI? § Measuring product owner value by number of backlog elements…and
doing ROI?
Based on some sort of automation introduction? Of course not. It’s insane.
§ Then why do we do it with tests…and testers?
Copyright © 2012 RGCG, LLC 13 13
14 Copyright © 2012 RGCG, LLC 14
n We’re all using them, so they must be good and the approaches must be ‘correct’…right?
n However, there are challenges— q Training q Size q UI-centric q Cost q Can’t test everything (Back-end database code) q One-size q Developer / whole team usage
Traditional Test Automation Tools Just a quick stop…
Typical Deployment Level of effort
Copyright © 2012 RGCG, LLC 15
Initial design & Architecture
Initial Test Case Automation
Technology Disruption
Ongoing Maintenance
Typical Time & Budget Investment
Now it may sound…
n Like I’m opposed to test automation. Truly, I’m not. I think automation rocks! Or like I’m opposed to gaining a ‘return’ I’m not q Automation is clearly a foundation
element for increased quality and going faster – releasing more often
n But, this traditional model or approach of focusing on ROI…is sort of insane
Copyright © 2012 RGCG, LLC 16
What agile has exposed to me… wrt/Automation & Testing
n Testers vs. a Whole Team view q Build teams in ratios – developers vs. testers q Instead invest in cross-functional teams composed of the skills to design,
construct, test, and deliver excellent products
q Only testers write test automation q The entire team writes test automation and, oh by the way, can test as
well
q Only testers test the application q The whole team participates in writing / running automation, functional
testing, exploratory testing, regression testing, etc.
Copyright © 2012 RGCG, LLC 17
What agile has exposed to me… wrt/Automation & Testing
n Traditional vs. Open Source Tools
q Instead of leveraging singular, UI-centric tools q Leverage multiple tools at all-tiers of your application stack
q Higher cost q Lower cost and a sense of “Community”
q Developers won’t use them; costs q Everyone is leveraging the same tool-sets; AND integrating them in
Continuous Integration and Continuous Deployment
Copyright © 2012 RGCG, LLC 18
What agile has exposed to me… wrt/Automation & Testing
n People are your primary value proposition
q Instead of looking at testers as a commodity q Consider good testers to be as valuable as any other team member
q Instead of thinking that testing is a by-rote exercise with little intellectual pursuit
q Testing is a profession. Testers are valuable craftspeople. Testing is an intellectual exercise.
q Instead of thinking the value is in the test cases q The value is in the testers (people) continuously reevaluating the right
things to test at the right time…for valuable feedback
Copyright © 2012 RGCG, LLC 19
Copyright © 2012 RGCG, LLC 20
http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html
The NEW Test Automation Business Case n So, don’t
q Count pennies q Count heads q Count cycles
n Work as part of a TEAM q Focus in on testing continuously – just the right things at just the
right time q Providing feedback, transparency, and reaction q Allowing your teams time to learn and improve q Solving your customers’ problems; delivering value
Copyright © 2012 RGCG, LLC 21
22 Copyright © 2012 RGCG, LLC 22
n Run more tests, faster, unattended, continuous feedback, learning & adjustments q Yes, saving time. Now what to do with it???
n Drive your new VALUE by: q Early feedback for adjustments—risk avoidance, learning, solving
real problems q Having more time to continuously refine your test cases
(automated and not) q Having more time to focus on building Quality into your products q Creatively testing the application (Exploratory Testing, Test Idea
Generation, Test Design)
The NEW Test Automation Business Case
So, with the additional time… INVEST in your testers & quality ü In building In Quality ü In attacking the customers REAL problems ü In better testing ü In more balanced automation ü In team training; in slack time ü In better collaboration ü In doing more than thought possible ü In creative solutions ü In adapting to new technologies ü In having FUN
Copyright © 2012 RGCG, LLC 23
How do we “Sell” this change? n You don’t sell it from a testing perspective. Nor from a bean-counting perspective.
n You allow the results to begin speaking for themselves as we begin focusing on and talking about— q How one builds truly innovative and creative software teams q That understand and solve customer problems q With simple solutions that fundamentally work q And do so iteratively & quickly, while nimbly adjusting to
customer feedback
Copyright © 2012 RGCG, LLC 24
But Bob…
n That doesn't work in the “Real World”!
n My boss would never go for it without ROI and hard justification
n Testing IS a commodity
n We’ve done automation perfectly – seeing none of the issues/risks you’ve pointed out
Copyright © 2012 RGCG, LLC 25
But Bob…
n Think about what I’m saying; taking the time and…
q Invest in people q Invest in better quality & testing q Invest in better product solutions to customer problems q Invest in improved workflow and delivering high customer value
WOW – how crazy is that? n Meet in small groups of 2-3 (pro / con) and chat for 10
minutes about the “sides” and the possibility for change? n Then we’ll debrief…
Copyright © 2012 RGCG, LLC 26
Copyright © 2012 RGCG, LLC
Wrap-up
§ Hope we challenged your existing assumptions a bit § Inspired you to change your view towards Automation
ROI and investment § What did I miss?
§ Final questions or discussion?
Thank you!
27 27
Contact Info
Bob Galen Principal Consultant,
RGalen Consulting Group, L.L.C.
Director of Agile Solutions, Zenergy Technologies,
Experience-driven agile focused training, coaching & consulting
Contact: (919) 272-0719
[email protected] [email protected]
www.rgalen.com
Blogs Project Times -
http://www.projecttimes.com/robert-galen/ Business Analyst – BA Times -
http://www.batimes.com/robert-galen/ My Podcast on all things ‘agile’ -
http://www.meta-cast.com/
Scrum Product Ownership – Balancing Value From the Inside Out published by RGCG in
2009.
28 Copyright © 2012 RGCG, LLC 28