ConfidentialPA12013-12-131
The Value-Adding TesterHow a Tester Adds Value to an Organization or Project
ConfidentialPA12013-12-132
Introduction
▪ The Value-Adding Tester adds value to the project or organization
▪ The Value-Adding Tester understand and reduces costs
▪ The Value-Adding Tester does not detract value
▪ To be a value-adding tester it is required to continuously evaluate value and cost
▪ How does a tester add and detract value, and what are the costs?
ConfidentialPA12013-12-133
How to Add Value - Overview
Fixed DefectsInformation to Stakeholders
Development Support
Mandatory Tests
ConfidentialPA12013-12-134
Fixed Defects
▪ One of the major ways a tester can add value is by finding defects in the product which are actually fixed at some point during the product life cycle
▪ The value of a defect found can be quantified by estimating how the defect would impact sales, and if the customer returns the product because of the defect
ConfidentialPA12013-12-135
Information to Stakeholders
▪ Another way to generate value for a tester is to provide stakeholders with information and material for tollgates, milestones and decision points
▪ The value is harder to quantify but can be divided into two parts:
▪ Helping stakeholders taking informed decisions
▪ Allowing stakeholders to feel more confident in their decisions
ConfidentialPA12013-12-136
Mandatory Tests
▪ In many cases different governing bodies require that certain criteria are met for a product to be allowed to be sold
▪ Meeting these criteria often require mandatory tests that must be executed according to strict requirements
▪ Specific customers can also have mandatory requirements that must be tested to be able to sell to those customers
▪ The value a tester provides by executing these tests can be quantified by comparing how much it would cost to outsource this to an accredited outsourcing partner
ConfidentialPA12013-12-137
Development Support
▪ A tester can also provide value by increasing the efficiency of the development team
▪ This can be done by providing a robust, easy-to-use test framework, or by creating and maintaining automated integration and regression test suites which the developers can use
▪ The tester can also offer support by helping with unit test plans and strategies
▪ Yet another example could be pushing for and educating in testability, which can also drive efficiency
ConfidentialPA12013-12-138
How to Detract Value - Overview
Irrelevant Defects
Irrelevant Information
Inefficient Test Tools
Low Quality Reports
ConfidentialPA12013-12-139
Low Quality Reports
▪ Testers should not create defect reports that
▪ Are hard to understand because of bad language
▪ Are duplicates of already existing defect reports
▪ Lack information that is critical to understand the defect
▪ Lack critical attachments such as log files or screen shots
▪ Etc.
▪ This will lead to increased report handling and analysis effort for developers
▪ If testers create reports which cannot be understood due to improper structure, language, formatting, etc., this also costs additional analysis effort for stakeholders
ConfidentialPA12013-12-1310
Irrelevant Defects
▪ When a tester reports a defect this has to be analyzed, prioritized and handled in different ways
▪ If the defect is not fixed, and the information stored in the defect is not used by stakeholders, submitting the defect actually detracts value instead of adding
▪ Many irrelevant defects could however together provide valuable data by revealing trends or problem areas
▪ The value detraction can be reduced by handling less relevant defects differently than higher priority issues
ConfidentialPA12013-12-1311
Irrelevant Information
▪ When testers provide reports to stakeholders, the information presented must be beneficial and help the stakeholders to take required decisions
▪ Presenting data that is not relevant or even misleading is not only not valuable, but can be very costly
▪ Burying key information in a mountain of data
▪ A report that states “99 % Pass Rate” as the main conclusion, when the remaining 1% represents critical quality issues in the product, will not help stakeholders make the correct decisions
ConfidentialPA12013-12-1312
Inefficient Test Tools
▪ The cost that testers impose on stakeholders with inefficient tools
▪ When testers use tools that impact others, such as developers, project managers, or other stakeholders, in a negative way, this detracts value
▪ One example could be inefficient test frameworks or automation tools which cause developers to write inefficient tests
▪ Another example could be a reporting tool which is difficult for stakeholders to extract reports from, or the tool generates reports which take time to analyze and understand
ConfidentialPA12013-12-1313
Costs - Overview
Test Design & Execution
Test Planning Test Reporting
Test Tools & Frameworks
Administrative Overhead
ConfidentialPA12013-12-1314
Test Design & Execution
▪ Designing, executing, maintaining and porting tests
▪ Execution Effort
Exploratory Test Scripted Test Automated Test
Automated Test Scripted Test Exploratory Test
▪ Design Cost
ConfidentialPA12013-12-1315
Test Planning
▪ Planning test activities and creating corresponding test artifacts such as test plans
▪ Setting scope for different test activities
▪ Risk analysis as impact to scope selection
▪ Probability of failure▪ Technical risk analysis
▪ Impact of failure▪ Business risk analysis
ConfidentialPA12013-12-1316
Test Reporting
▪ Creating test reports
▪ Test result metrics
▪ Qualitative summaries/product stories
▪ Creating defect reports
▪ Analysis of automated test results
ConfidentialPA12013-12-1317
Test Tools & Frameworks
▪ Creating new tools and test frameworks
▪ Cost of buying test tools and frameworks
▪ Integrating new tools
▪ Maintaining tools and frameworks
ConfidentialPA12013-12-1318
Administrative Overhead
▪ How much time does the tester actually spent on design, analysis, execution, tools and reporting, and how much time does the tester loose to administrative overhead?
▪ Getting the right software artifacts for test, and understand what to actually test
▪ Coordinating between testers, developers, and organizations
▪ Non-value adding meetings, etc.
ConfidentialPA12013-12-1319
Net Value of a Tester
Cost Value DetractionValueNet Value
ConfidentialPA12013-12-1320
How do we increase value and decrease cost and value detraction?
▪ We want to maximize value gain, and minimize costs and value detraction
▪ How we do this depends heavily on context
▪ However we can still give some general guidelines
▪ Of course there are many other ways to reduce costs and add value, but these are some suggestions
ConfidentialPA12013-12-1321
How do we increase value and decrease cost and value detraction?
▪ Evaluate test artifacts and remove the non-value adding ones
▪ 10-minute Test Plan – don’t create extensive test plans that no one actually uses or updates
▪ Very detailed scripted test cases cost more than they add value most of the time
▪ Test Strategy documents that no one reads or uses should not be created at all▪ However the discussions which lead to the creation of the documents
are still valuable and important to have
ConfidentialPA12013-12-1322
How do we increase value and decrease cost and value detraction?
▪ Evaluate tools among all different stakeholders to secure that they are efficient
▪ Even if a tool is easy to use for some purposes, it could impose large costs for other stakeholders to use
▪ A test administration tool that is easy to record data in, may be very costly to generate reports from
▪ Evaluate the actual value gain of the automated test framework
▪ Don’t just calculate the execution effort saved▪ Many different costs: Design, Analysis, Maintenance, Porting
▪ Actual gains: Defects Found, Mandatory Tests, Information and Support to Developers
ConfidentialPA12013-12-1323
How do we increase value and decrease cost and value detraction?
▪ Secure that the information and defects testers report is what stakeholders actually need
▪ Defects must be fixed
▪ Information must be used
▪ Make sure that testers actually work with testing, and not spend their time on everything else
▪ Let testers list what they spend their time on, and try to reduce administrative overhead and let the testers actually work with test
ConfidentialPA12013-12-1324
How do we increase value and decrease cost and value detraction?
▪ Proper risk analysis to support scope selection
▪ Risk-based testing
▪ Should be used when the reduction in cost for test execution is larger than the increase in cost for test planning that the risk analysis adds
▪ Efficiency gain of risk-based testing is very dependant on the complexity of the system under test
▪ Show the cost that the system complexity adds to testing – this can drive actions to reduce the complexity of the system
ConfidentialPA12013-12-1325
Conclusion
▪ The Value-Adding Tester continuously evaluates what value is added, what value is detracted, and the costs for providing this value
▪ There are many ways to add value, but there are also many ways to detract value and either way there is an associated cost
▪ Understanding the values and costs is critical to become efficient
▪ Understanding must be supported by actual metrics, and not only gut feelings