Embed Size (px)
Citation preview
Introduction
Quality. It’s something key to our business, indeed, any business. It’s a crucial attribute of any product, whether that product is a device, a piece of software, an IT infrastructure or service. A product that lacks quality will fail in the marketplace.
Delivering quality is incumbent on those involved in the product development process and those involved in the associated testing. In this series of articles, we will be looking at how testing processes contribute to product quality and how it is crucial that these processes are regarded as an investment, and not a cost. To set the scene, in this opening article we will look at what we mean by ‘quality’ and some of the other terms that comprise the testing professionals’ vocabulary. We will ask the basic question ‘why invest in quality?’ before moving on to take a top-level exploration of some of the key criteria that define quality, leading us to an appreciation of why testing for quality really needs to be considered as a key investment.
White Paper: Investing in Quality
Why Investing in Quality is Important for your Business
In this first of a series of articles we’ll look at what investing in quality means and why:
• It is key for an entire organization to acknowledge the value and importance of Quality Assurance & Control. • An investment in Quality Assurance and Control can (and must) be considered as cost savers for the overall project cost. • Quality Assurance and Control should be applied to any development project, regardless of methodology or products under development.
What is Quality?
Let’s be honest. No organisation sets out to make a poor-quality product or service. Sadly, though, we have all seen products or experienced services that fail to meet our reasonable expectations. For example, a piece of newly released software that acts erratically; a TV set top box inexplicably ‘crashes’ when certain commands are entered or a phone app performs unpredictably. With all reputable companies testing their products before release, how can this happen? Why does this happen? And what can we do to prevent it from happening? These are questions we need to ask as we explore not just how we invest in producing quality products and services but how we effectively invest. So where are we coming from? Eurofins Digital Testing is a testing company. It works with companies in the technology sector helping them develop products and services that are robustly defined, effective, measurable and well received by the intended end users. Our teams and experts work with a vast number of industries with companies dealing with digital transformation. Through experience we are well placed to discuss both qualitatively and quantitatively why delivering quality is an investment, not a cost. Before we go any further, though, let’s start by defining some of the key terms we will be using throughout this series of articles.
QualityThe definition is often debated, even by many QA professionals, with some suggesting it should not be considered an empirical measure at all, whereas others will advise the meaning can vary according to context. Some apply a more naïve definition that suggests quality as merely ‘the lack of errors’ or, in the more specific context of software ‘the lowest incidence of bugs.’ Let us be guided by ISTQB, who provide a more formal definition, and the one that we will employ here:
Quality is: “The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations.”
TestingAnd what of Testing? Again, a term that many people understand in more general terms, but which needs, in this context a more solid definition:
Testing is: “The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of a component or system and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.”
The Cost of QualityThis leads us to describing what we mean by ‘the cost of quality’. The Cost of Quality (COQ) can be defined as a methodology that allows a company or organization to evaluate how much of their resources are being used on:
1. Activities that prevent poor quality2. Activities that can effectively appraise the quality of the organization’s products or services3. Appraise costs resulting from internal and external failures. Armed with this information an organisation can quantitatively determine the savings - or potential savings that can be gained through process improvements:
• Prevention costs• Appraisal costs• Internal failure costs• External failure costs
It is useful to summarise these here:
Prevention Costscosts incurred avoiding, or preventing, quality problems. These range from the costs associated with maintaining a quality management service through quality planning and maintaining a quality system. From a personnel perspective, costs associated with training staff also fall into this category.
Appraisal Costscosts associated with monitoring and measuring any activities that are related to quality. They can include everything from goods inwards costs through quality audits and any costs associated with monitoring or rating suppliers’ products and services.
Internal Failure Costs‘internal’ here relates to the organisation and the costs to failures that occur in-house. For example, correcting defective materials or processes, analysing causes of failure costs from scrapping defective units.
External Failure Costscosts associated with the products and services once they are in the hands of the customer. Think repairs, warranty claims complaints processes and the return or recall of faulty products. Consider also how external failure costs can trigger even greater costs (which are ultimately business losses) such as reduced revenue per user, customer churn and, less tangibly, reputational damage. There could also be imposed costs due to regulatory or standards compliance violations.
We will revisit these in a later article but it is also worthwhile looking figuratively at how the Cost of Quality in respect of the cost of non-conformance (internal and external failure costs) and the cost of conformance (quality assurance and testing) can change between the default (or conventional) situation and that in a situation where there has been investment in QA and testing. This is shown illustratively in the figure here. It should be clear that, although there will be some variation in the amounts in each individual case, investing in quality will lead to a substantial decrease in overall costs.
Figure 1: Cost of Quality compared in default situations and
where QA/Testing investment has been made
COST OF QUALITY
Cost of Conformance (Quality Assurance & Testing)
Cost of Non-conformance (Internal & External Failure Costs)
Default Investment in QA & Testing
Why Invest in Quality?
Have you ever heard someone say “Let’s not invest in quality.” No. No company, and no testing manager will ever say that. But often that is the result if testing a product or service against specifications is given a low priority and the relevant testing relegated from an intrinsic part of the development process to a round of box ticking at final project sign off.
Companies that do this either consciously or unconsciously are not really investing in testing for quality so much as making it part of the ‘developmental due process’. It really does then become a case of ticking several ‘test passed’ boxes before describing the project as ‘passing all tests’.
First, the product or service could successfully pass all testing requirements and be waived through for release. Success in the marketplace will now depend on how thorough that testing was, and whether all eventualities have been captured. Often this is not the case. Not all situations the product might meet in the marketplace will be captured. The net result is that given the marketplace is very unforgiving, customer confidence will be quickly undermined, impacting ongoing sales, reputation and future business.
Second, the testing process can identify that the product or service has failed to meet one or more of its objectives. The product is sent back to the development team, or the service plan revised for the failings to be fixed. A simple operation perhaps, but one that:
• Takes time – affecting product release schedules and potentially other project delivery dates• Costs money – affecting project expenditure which may not have been originally budgeted • May introduce other defects – further impinging on timescales and budgets• Could introduce ‘project panic’ where deliverables are rushed back into the production environment without being fully tested.
Such delays can also affect customer confidence and perception, when the product or service does not hit the marketplace as planned and potentially with a reduced scope.
This is not the case in many companies, however, where there is a culture of quality and QA that pervades the organisation at all levels. When all team members involved in a project, not just those involved notionally in ‘testing’ are involved, potential problem areas and bugs can be identified and fixed early in a development cycle and often before they have the chance to become formal ‘issues’. In solving these early on the cost of correction is lower – often minimal – and the risk of late or poor-quality delivery is minimised.
QUALITY
Scope
Schedule Cost
Project Management Triangle Quality in a project, is determined by certain constraints. This is captured by the Project Management Triangle, as shown below:
Figure 2: Project Management Triangle: Definition of
Constraints
The Project Management Triangle contends that: • Quality is constrained by a project’s Budget, Deadlines and Scope • A project manager can vary the constraints, trading off one against the other• To maintain quality, when one constraint is varied, others must be to accommodate
For example, a project may be completed faster by increasing the allocated budget or cutting the scope. Similarly, increasing scope may require equivalent increases in budget and schedule. However, if the budget were cut without adjusting schedule or scope, the result would be lower quality. An important part of our contention that companies should be ‘Investing in Quality’ is the relationship between cost and quality as cost is inevitably the most significant aspect in most, if not all, projects.
If we now take the ISTQB’s definition of the Cost of Quality, as described above, Costs will be: • Development costs & other project related costs • Quality related costs (as touched on earlier): o Prevention costs, o Appraisal & testing costs, o Internal failure costs and external failure costs. Shifting costs to investment in prevention and appraisal costs will have a positive effect on quality and on the overall costs. With less being needed to be spent on internal and external failure costs the cost of the products and services can be better contained with obvious benefits when these are passed to the customers in terms of lower prices.
Constraint Defined as
Scope The size of the feature set, the complexity, the scale and type of deployment Cost Principally, the amount of resource invested in the project, both staff and hardware, software, infrastructure and also including overheads Schedule Determined by the required release schedules / time-to-market of feature & maintenance releases.
Positioning in function of customer project constraints
When we are partnering with companies to assist them with their quality assurance and testing programmes we need to properly assess the project in terms of the constraints in the project management triangle of scope, cost and schedule. Each of these will be subject to different pressures which need to be considered.
Pressure on Costs and Budgets No matter the business sector, cost control is always an important factor. Competition and profitability will put pressure on reducing costs and costs related to quality assurance and testing are often particular targets. What budget has been made available for testing can also be applied to the wrong items and at the place in the development timeline.
Figure 3: Relative cost to fix defects in software, based on
time of detection
Pressure on Schedules (and time to market) As in respect to costs, increased competition puts pressure on development schedules, driving them downwards. It is important that the timelines for the quality assurance and testing processes are given due prominence and where optimisation is required it is done so by employing appropriate process efficiency measures. Realising how the time pressures affect companies we have introduced optimized quality assurance and testing processes to accelerate time-to-market of new products & services. This includes
• Risk-based testing • Controlled test processes (incl. use of relevant tools) • Test Automation • Efficient manual test execution • Exploratory testing approaches• Reusable test assets• Blended manual/automated testing.
Pressure on Project Scope The need to be innovative and incorporate advances in technology inevitably means that new or updated products have a rich feature set. This means that the technical scope and supporting environments and data of the quality assurance and testing processes must match the scope and complexity of the new device or service under test.
In practice this has involved the introduction or operation of enhanced quality assurance and testing solutions that effectively accommodate the extended or new feature sets.
RELATIVE COST TO FIX,BASED ON TIME OF DETECTION
Requirements/ Architecture
Source: National Institute of Standards and Technology
0 x
5 x
10 x
15 x
20 x
25 x
30 x
CodingCoding Integration/Component
Testing
System/Acceptance
Testing
Production/Post-Release
The False Economies – and costs – of Not Investing in Quality
Off-shoring is another example where cost savings are made but the consequential effect on quality can be disproportionately large and is something that particularly affects software and service offerings. Enticed by lower costs the omens are often good, as initial work is successfully conducted. However, iterative work is often more laborious and customer satisfaction can often nosedive, leading, for example, to a churn in service customers that negate the savings.
We will have more to say about this, too, in future articles. Sadly, approaches to testing of this form are still all too common. The testing and QA process has not been given the prominence it deserves and may, consciously or otherwise, be seen by management as a necessity, but one that comes with a cost. A cost that needs to be minimised.
Testing and QA as an Investment
So, how can we turn this situation on its head so that testing and QA are perceived as investments? It calls on a structured approach that first acknowledges that testing and QA are not discrete activities divorced from the main development and are equally important contributors to product quality. More significantly they need to be seen not as cost centres, but as ways of maintaining – or even reducing – overall product costs.
Those involved in product and service development and QA will have the same goal: delivering a quality product on time and on budget. Both teams need to work closely together in a collaborative manner, although both need to remain independent in their decision making, based on their respective KPIs.
The QA function in a business has undergone significant change over the last few years, where we have seen traditional assessment methods give way to the widespread adoption of agile methodologies and DevOps practices. Development cycles have become more rapid as both competition and rapidly evolving technology have applied pressure to developers. The positive impact of these changes is that they have added to the rigour in which testing can be performed and helped develop a more structured approach. It has also seen QA and testing generally become more firmly embedded as an intrinsic part of any project development plan.
Outsourced Testing A concern we hear raised often is that QA can incur costs because of the need to have an expert team in place to deliver the service appropriately, even though the scope of this need can vary significantly throughout the project. This has led to the use of Managed Testing Services, delivered by partner companies. See ‘What is a Managed Service’ for more information. This also overcomes the problems with having the right skill base available for any project and scaling this to actual need.
What is a Managed Service?Managed services, delivered by third party partners, are an excellent way of scaling your operation to accommodate a testing and QA regime without the need to invest heavily in building up and maintaining in-house testing expertise. A testing managed service partner will help you develop a vigorous testing regime, with the resourcing manpower delivered by the partner only when and as needed by the project. You get a scalable resource only when needed and without an ongoing cost overhead.
Investing in Quality: The Story Continues
There is a widely held perception that in almost any business higher quality means higher costs. That can be quantified in an obvious sense by using higher quality – and more expensive – raw materials, employing more skilled staff members, though to spending longer refining a process or service specification. And many more things besides.
However, we have seen, in a qualitative sense, through judicious use of testing and QA procedures we can streamline the delivery of quality, reducing the associated costs and making the processes that help that delivery into an investment, not a cost. Of course, to successfully ‘invest in quality’ we need to delve deeper into the processes and methodologies associated with testing and QA. That’s what we’ll be doing in future posts in this series. We invite you to return / register / click to follow these future instalments.
Coming soon We’ll be looking in more detail at investing in quality in these following posts.
The Importance of QAWe’ll examine the cost of defects and learn more about CoQ, the cost of quality, along with shift left/early phase testing. We will also look at new development methodologies and their impact.
Achieving Cost Efficiencies in QAHow, by embracing QA and exploiting new methodologies we can cut the costs associated with QA processes. We look at the impact of test automation in in both stand alone and blended testing strategies.
How to Budget for QAWe’ll examine a breakdown of a typical QA organisation, budgeting considerations per for different solutions (such as resourcing and managed services) and how advisory services can help organisations capitalise on QA skills.
