Upload
maude-flowers
View
218
Download
3
Tags:
Embed Size (px)
Citation preview
© Company Confidential www.itcinfotech.com
Business Case for Test
Automation
S.JanardhananChief Technology Officer
ITC Infotech India Limited
Business Case for Test
Automation
© Company Confidential www.itcinfotech.com
Agenda
• Test Automation – Myths and Reality
• Return on Investments • Key take outs and summary
© Company Confidential www.itcinfotech.com
Test Automation – Myths and Reality
© Company Confidential www.itcinfotech.com
Test Automation Myths
• Test Automation is simple, every tester can do it
– This myth is promoted by the tool sales people. They are trying to promote the following test automation process:
• Record the script• Enhance the script by adding functions and data
driving• Run the scripts• Report results
– Under the influence of this myth a QA manager can proudly report: All our testers are developing test automation.
© Company Confidential www.itcinfotech.com
Reality - Test automation is a software development task
• Automation should be designed, developed and tested
• You need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development.
• Test automation standards should be developed
• Automated test components are assets that should be treated like application source code, unit tested, integration tested and performance aspects considered.
© Company Confidential www.itcinfotech.com
Myth #2 – Commercial test tools are expensive
• Under the influence of this myth some companies, especially the small ones:
– Try to develop their own test automation tools– Use scripting languages like Perl and Ruby– Use shareware test tools– Do not consider test automation at all
© Company Confidential www.itcinfotech.com
Reality – Commercial tools are cheap
• Per seat license for most expensive automation tool is $8K
• This tool will be used for 5 years.
• Maintenance/Support fees are 20% of tool cost or $1,800 per year
• The cost of this tool is $8K/5+$1,800 = $3,100 per year
• The automation developer cost with overhead is $100K per year
• The cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant
© Company Confidential www.itcinfotech.com
Commercial Tool Benefits:
• Customer support. Many of the open source tools come and go with little to no support
• Most commercial tools are constantly being updated as technologies change
• Most commercial tools usually have more functionality (QTP can test various GUI applications: Web, .Net, Java, VB, C/C++, PowerBuilder, etc. vs. WATIR – Web only)
• Commercial tools usually have a large community of users, which translates into better availability of qualified resources
• Commercial tools require less advanced programming
• More test automation frameworks are available for commercial tools
• Commercial tools are integrated with Test Management tools which make reporting and execution much simpler.
© Company Confidential www.itcinfotech.com
Return On Investment (ROI)
© Company Confidential www.itcinfotech.com
Classic ROI Calculation
ROI = BENEFIT/COST
Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost
ROI = (Manual Testing Cost - Automation Cost)/Automation Cost
Looks right, doesn’t it?
© Company Confidential www.itcinfotech.com
Problems with Classic ROI Calculation
• You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.
• You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.
© Company Confidential www.itcinfotech.com
Automation Real ROI
• ROI value IS NOT the value of Automation vs. Cost of executing these tests manually
• Automation ROI value IS the benefit of this type of testing, and it can be:
• Reducing Time to Market / Execution Time• Increased Test Efficiency (Productivity)• Increased Test Effectiveness & test coverage• Decrease in test defects escaping to production • Improved test repeatability
© Company Confidential www.itcinfotech.com
Reduced Time to Market
• Can get a greater market share
• Makes people available to work on other projects
• Higher margins, if no competitive products are currently available
© Company Confidential www.itcinfotech.com
Productivity and Effectiveness
• More testing gets done faster, increasing the odds of finding defects
• Defects found early have better chances to be fixed• Manual Testers can concentrate on clever ways to finding
defects, instead of typing test inputs and verify output. • About 7% of bug fixes create new bugs, sometimes in
already tested parts of the system. With automation you can rerun tests for those modules. This almost never happened when testing done manually.
© Company Confidential www.itcinfotech.com
Manual VS Automated Testing Example
Test Cycle Manual Execution Time (days) Automation Execution Time (days)
1 7 30.5
3 21 31.5
5 35 32.5
7 49 33.5
9 63 34.5
• There is a suite of manual test cases that have been deemed as candidates for automation. When run manually, the execution can start immediately (ITC = 0 days). It takes the manual testing team seven business days to execute all of the test cases (TEC = 7 days). The Total Testing Time will always be a linear factor of seven days.
• If the test cases are to be automated it will take 30 business days to complete (ITC = 30 days). Once the test scripts have been written it takes half of a business day to complete the tests with the necessary hardware resources (TEC = 0.5 days).
© Company Confidential www.itcinfotech.com
ROI summary
• Each project requires different types of automation - there is no easy formula available to calculate ROI
• Performing ROI calculation can help to determine upfront what type of automation, what level of skills, what tools will be required.
© Company Confidential www.itcinfotech.com
Thank You