Embed Size (px)
DESCRIPTION
Testing has evolved. From being an activity that “just had to be done” to a profession. And from just a profession to a fizzing profession with its own methods, tools and techniques on test execution, management and even on strategic levels. Still, a question often posed to the test professionals is: “Why does testing cost so much money?”.
Citation preview
Identifying the benefits of testingRené Tuinhout, Richard Janssen
TESTING
0202
Testing has evolved. From being an activity that “just had to be done” to a profession. And from just a profession to a fizzing profession with its own methods, tools and techniques on test execution, management and even on strategic levels. Still, a question often posed to the test professionals is: “Why does testing cost so much money?”.
The perception of testing is that it’s expensive. While we, the test profession, have become quite versed in explaining why testing has a certain cost, often we have been not been able to change this question into the far more interesting question: “What does testing contribute to the stakeholders?” Or, in other words “What costs does testing prevent?”.
By answering this particular question to our stakeholders we would rapidly change the perception of testing as a necessary evil into that of testing as a contributing factor in the production process.
03
And we do have the tools and knowledge available to answer this question. Some attempts have even been made: For example, it´s possible to create a ‘history database’, a database containing data regarding previous projects and failures, and their costs. By analysing the database for failure patterns and connected costs, failures occurring often and leading to grave losses can be identified. Then, tests can be designed to try and detect those failures before they end up in production. Basically the rationale used boils down to:“after the last project, X failures of type Y occurred in production, costing Z. In this project, by testing we found A Y-type failures, so we saved our organisation A*(Z/X)”. However, setting up such a database takes time. And are failures found in different applications really that alike? Can they really be compared to one another that easily? Too often, these questions lead to lengthy and unresolved debates.
This article suggests another way of calculating the benefits of testing. One, in which the costs saved are not directly related to failures (expected to be) found, but to risks. It adds to a technique we as testers use already: risk and requirement based testing.
What we as testers have been doing in the past years, is finding out the risks involved in applications and linking a test priority to that risk and the associated requirement(s). Is the risk high? And is the requirement(s) an absolute must? Then it is something that must be tested!
The process of finding risks and prioritising them has evolved in the years past1. Before, it was common practice to interview one or more stakeholders and discuss the risks involved in the application. This has evolved into the practice of interviewing stakeholder management not regarding specific risks, but regarding the risks on a conceptual level: All management involved (stakeholders) together create a “risk thermometer” specific for their organisation. With risk being defined as probability times impact, management is asked which high level impact(s) they can envisage for the application(s) under test. Being smart testers, we do not only ask for impact in terms of money, but also in terms of loss of image and non-compliance to law or regulations. Stakeholder management will, after some discussions, come up with an “impact-thermometer” (Figure 1).
Loss of Money
*Total loss of >=M€5
* One customer affected and displeased*Total loss of >k€1 and<=k€10
* In two main newspapers
Board of diectors in jail
* warning is received
None
* In the newspapers A,B,C,D or E
* All potential clients and existing clients affected
No significant loss None
*Loss of >= k€100 per incident
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
......
Loss of imageNon-compliance to
law/regulations
Impact
1
2
3
4
5
6
7
89
10
Figure 1: The impact-thermometer, an example.
1This article will focus on the risk part of risk & requirement based testing.
04
By combining the probability and the impact thermometers the risk thermometer is produced (figure 3).
After having created the impact-thermometer, stakeholder management is asked to create the probability thermometer in much the same way. Not only the frequency of certain actions should be considered in this activity, but also the urgency: the probability of a certain action occurring shortly after going live.
For example: An end-of-year-run that’s executed once a year on January 2nd has a very low (annual) frequency, which would suggest a low score on the probability-thermometer. However, when the go live is scheduled for December 27th, one of the first things that’s going to happen after the go live is this end-of-year-run. Which should increase the score on the probability-thermometer (Figure 2).
Frequency
* Occurs >= 10 times each day
* Occurs once a week
* Occurs once a year
* will occur within 1 week of going live
* will occur within three months of going live
* will occur more than a year after going live
UrgencyProbability
5
4
3
2
1
Risk
10
9
8
7
6
5
4
3
2
1
Loss of Money
*Total loss of >=M€5
* One customer affected and displeased*Total loss of >k€1 and<=k€10
* In two main newspapers
Board of diectors in jail
* warning is received
None
* In the newspapers A,B,C,D or E
* All potential clients and existing clients affected
No significant loss None
*Loss of >= k€100 per incident
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
......
Loss of imageNon-compliance to
law/regulations
Impact
1
2
3
4
5
6
7
89
10
Frequency
* Occurs >= 10 times each day
* Occurs once a week
* Occurs once a year
* will occur within 1 week of going live
* will occur within three months of going live
* will occur more than a year after going live
UrgencyProbability
5
4
3
2
1
Figure 2: The probability-thermometer, an example.
Figure 3: Combine probability and impact to risk.
05
Based on the risk-thermometer, discussions with stakeholder management can commence, leading to a MoSCoW division of the risk-thermometer: Which risk-rating would, for you as the stakeholders, lead to a “must-test” risk? And which to a “should-test”? (figure 4).
Having created the risk thermometer with management, the risk perception of management has been visualised and made measurable. Consequently, an organisation-specific model has been created: A model indicating the managements view on risks on a conceptual level.
This model can be used to create an inventory of the actual, real, risks in the application under test itself. This inventory is created in discussions with stakeholder staff, the actual users2 of the application under test. In a workshop risks are identified and, using the probability- and impact-thermometers, the degree of risk can be calculated for each identified risk. This results in an overview of identified risks, and a degree of risk per each risk identified (figure 5).
Abiding by this process has enabled us as testers not only to prioritise tests better and in alignment with stakeholders and stakeholder management views. It also enabled us to detect possible threats to the test process itself in an early stage: If, for example, the majority of the identified risks are in the must test area, and the time provided for testing is very limited, insight to stakeholder management can be created (using the thermometers shown), demonstrating either that more testing time or testing staff is needed. Or that stakeholder management might want to reconsider their MoSCoW-division of the risk thermometer. After all, by shifting the risk-ratings, some risks end up in other MoSCoW-categories, leading to another test-effort. (In the example in figure 6, stakeholder management shifting the Must-test boundary upward will lead to less must-test test cases, but to more should-test test cases3.)
Based on the organisation specific risk model created by management, and based on the specific risks prioritised according to the model created by staff, testing can now be prioritised and managed.
But, there’s another consequence of using the process described above: By creating test cases in such a way that it’s traceable which risk(s) are associated with a test case, now reporting on risk coverage becomes possible: Instead of reporting “test case XYZ failed” as was the case in the early years of testing, nowadays “risk ABC isn’t (fully) covered yet”4 can be reported. This way of reporting on risk coverage is, compared to the reporting on failures, far more insightful for testing’s stakeholders.
Risk
10
9
8
7
6
5
4
3
2
1
Must test
Should test
Could test
Won’t test
Figure 4: MosCow division in the risk thermometer.
Figure 5: Identified risks on the risk thermometer, represented by red dots. Risk 17 shown as an example.
Figure 6: A shift in the risk-model, decided upon by stakeholder management.
Risk
10
9
8
7
6
5
4
3
2
1
Must test
Should test
Could test
Won’t test
2By actual users all (future) application users are ment, e.g. front end users, maintenance staff, management in a user-role. Depending on the test level to execute, development staff could, of course, also be involved in the workshop.
3The test plan, of course, should define which techniques (and which test coverage and depth) should be used for Must-test test cases, for Should-test test cases etc. For this article it’s assumed that testing a Should-test test case requires less effort than testing a Must-test test case (as is usually the case).
Risk
10
9
8
7
6
5
4
3
2
1
Must test
Should test
Could test
Won’t test
Risk 17: Calculation of fee failure. Affects all customers,reaches main newspapers.
06
However, even creating this insight does not provide an answer to the question posed at the outset of this article:“What costs does testing prevent?”.
‘Adhering’ to the method described above, a short answer to the question could be:”In the impact thermometer, a ‘loss of money’-column is defined. So when a test case fails, find the risk related to the test case, then find the loss of money related to this risk, and report it as costs prevented by testing.” And basically, it is that simple!
Unfortunately, this simple answer often leads to discussions: It will be argued the test case failed and therefore indeed the risk is not covered, but that the failure is not that severe. Suggesting not the entire risk would actually have occurred when the failure would have ended up in production. Furthermore, the loss of image and the non-compliance to law/regulations often is not defined in a monetary way. This can lead to discussions about which costs were prevented by detecting a failure related to the risks defined in these ways.
To avoid these discussions, we propose to add a bit more information to the risks identified: The minimum and maximum cost of failure. While in discussion with stakeholder management, apart from creating the classic organisation specific risk model, in this model the minimum and maximum costs related to the risk classes identified should be included (figure 7).
A last suggestion: When minimum and maximum costs per failure detected are included in the defect
Loss of Money
*Total loss of >=M€5M€5
M€3 M€5
* One customer affected and displeased*Total loss of >k€1 and<=k€10
* In two main newspapers
Board of diectors in jail
Total value oforganisation(wash-out)
* warning is received
None
* In the newspapers A,B,C,D or E
* All potential clients and existing clients affected
No significant loss None
*Loss of >= k€100 per incident
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
......
Loss of image Minimumcost
Maximumcost
Non-compliance to law/regulations
Impact
1
2
3
4
5
6
7
89
10
Figure 7: The impact-thermometer including minimum and maximum cost, an example.
4It’s assumed test case XYZ failed because of a fault, not because the test case itself was flawed. This assumption will continue throughout this article: Whenever a failed test case is mentioned, it’s assumed it was a failure caused by a fault, not by a flawed test case.
07
reporting, this has two advantages: The stakeholder representative(s) related to the test project for deciding on what to do with defects found, are informed of the (mimimum and maximum) costs of a failure. This is generally viewed as a welcome extra input for the decision process. Furthermore, including the minimum and maximum costs per failure detected in the defect reporting allows the test manager to report on the benefits testing has had fairly easily.
In short: Including the minimum and maximum cost of a risk occurring will take some time in the early stages of a test project, when discussing the organisation specific risk model with stakeholder management and specific risks with stakeholder staff. However, it prevents (much more) discussions later on in the project about the costs involved if a risk occurs.
And, more importantly, it will allow us, professional testers, to finally report on the benefits of testing.
This model should then be used as a basis for the discussions with stakeholder staff in which the real risks are identified. For these discussions we again suggest not only identifying the actual risks, but also the minimum and maximum costs related to each risk identified (figure 8).
This will lead to discussions, especially regarding the costs of less measurable topics such as “loss of image” and “non-compliance to law/regulations”. Questions such as “how to measure this in a monetary way” will arise. However, experience shows these difficulties can be overcome in discussions amongst the stakeholders, leading to defined minimum and maximum costs the stakeholders agree to5.
Identifying these minimum and maximum costs in the early stages of the test project allows the test manager to report not only on faults detected or (via traceability to risks) on risks still not (entirely) covered. It also allows the test manager to report on the minimum and maximum benefits testing has had for the organisation. Not by reporting the benefits the test manager has identified him/herself (which would lead to heated discussions), but based on information provided by the stakeholders themselves.
Risk
10
9
8
7
6
5
4
3
2
1
Must test
Should test
Could test
Won’t test
Risk 17: Calculation of fee failure. Affects all customers, reaches all main news papers. Minimum cost €M 8, Maximum cost €M 12(costs like: damage control, clients leaving, no new clients, damages)
5And, sometimes, even to an organisation specific model defining loss of image and non-compliance –risks in monetary units.
Figure 8: maximum cost, an example.
LogicaTel: +31 (0)88 564 [email protected]
www.logica.com/testingCODE 3623 1011
Copyright © 2011 LogicaAll rights reserved. This document is protected by international copyright law and may not be reprinted, reproduced, copied or utilised inwhole or in part by any means including electronic, mechanical, or other means without the prior written consent of Logica.Whilst reasonable care has been taken by Logica to ensure the information contained herein is reasonably accurate, Logica shall not,under any circumstances be liable for any loss or damage (direct or consequential) suffered by any party as a result of the contents of thispublication or the reliance of any party thereon or any inaccuracy or omission therein. The information in this document is therefore providedon an “as is” basis without warranty and is subject to change without further notice and cannot be construed as a commitment by Logica.
Logica is a business and technology service company, employing 41,000 people. It provides business consulting, systems integration and outsourcing to clients around the world, including many of Europe’s largest businesses. Logica creates value for clients by successfully integrating people, business and technology. It is committed to long term collaboration, applying insight to create innovative answers to clients’ business needs. More information is available at www.logica.com.
Copyright statement
René Tuinhout is a Practice Lead for Logica Netherlands’ Testing & Quality Management practice. He´s a senior test advisor and test coach with over 14 years of experience in software testing. René advises organizations on test management, test change management, structured testing, and he manages test (change) projects. René tutors courses in Testing Techniques, test management and several other test related subjects and is a speaker on (inter)national conferences on Testing.
Richard janssen is a practice lead for logica netherlands’ Testing & Quality Management practice. Richard works on strategy development, marketing and solution development for logica’s t&qm business. Richard is a corporate generalist with a broad view on techno – economical trends, organizational change and business/technology integration. He has a degree in electrical engineering & computer science from university of twente and an mba from tsm business school.
