Embed Size (px)
Citation preview
Risk Mitigation: Fixing a Project Before It Is BrokenA comprehensive assessment of unforeseen risks in the project lifecycle can prevent costly breakdowns at the testing stage.
Executive SummaryMany IT projects have a reputation for exceeding budgets, missing deadlines, failing to realize expectations and/or delivering sub-par return on investment. Surveys and reports on the accept-ability of new IT systems reinforce the same problems and probable causes of failure; yet businesses, large and small, continually make mistakes when attempting to improve informa-tion systems, particularly investing in inappropri-ate or unworkable changes without proper con-sideration of the likely risks.
Planning projects, establishing requirements for change, selecting, implementing and testing suitable systems are important components of a superior business management process. Companies, however, often do not take adequate precautions during the project lifecycle to minimize failures at the testing stage. Consider the following:
• Our assessment of 134 publicly traded companies reveals that over half admitted to suffering a failed IT project in the past 12 months due to issues encountered during the testing stage. Such failures come at an extremely high cost.
• According to the Royal Academy of Engineer-ing and the British Computer Society, £3.5
billion are wasted every year on badly managed IT system deployments, only discovered during the testing stage.
• Only about 16% of overall IT projects can be considered successful; even conservative estimates put the costs of failure into the £20 billion range across the European Union (EU), in IT software development projects.2
As seen in Figure 1 (next page), the peak expen-diture of both time and money on IT support and development occurs at the testing stage. The costs of failure are not limited to the narrow con-siderations of budget and schedule. Companies do not pay adequate attention to conducting a proactive risk assessment of all projects during their entire lifecycle in an effort to mitigate and minimize the risk of failure at the testing stage.
Identifying and analyzing potential risks of failure during the project lifecycle is fundamental to avoiding issues at the testing stage, as well as massive rework. To do this, it is vital to set up an early-stage diagnosis. This approach includes analyzing projects throughout the entire lifecycle, from the requirements gathering stage through verification, when all potential variables and intermediate process steps are weighed in terms of risk. Taking this approach can help IT organi-zations understand and minimize the potential impact on testing activities.
• Cognizant 20-20 Insights
1
cognizant 20-20 insights | july 2012
cognizant 20-20 insights 2
In this whitepaper, we spell out our rigorous process for identifying and avoiding project issues that undermine many applications development projects. It also offers proven ways for organiza-tions to minimize development delays and costs, while building systems that deliver exceptional business benefits.
Test Risk Assessment ProcessCognizant Business Consulting developed a structured, proprietary process called the Test Risk Assessment for Project Improvement (TeRAPI), with the main objective of identifying and minimizing risks that can occur at the testing stage across the entire product lifecycle.
The process consists of three main phases:
1. Measure the business criticality.
2. Profile the risk level of projects.
3. Design a risk mitigation plan.
The purpose of measuring business criticality is to determine the importance of each business function. After all the business processes are evaluated to determine their criticality to the project, business functions can be prioritized to determine which ones require a contingency plan and which can be ignored and eliminated.
Each function, system, interface and third party can and should receive a risk rating. Probabili-ties are assigned to each failure that can occur
for each of these items. This means that an order function (risk rating = 5) could be impaired because of a business partner failure (probability of 60%) or could fail due to a system failure (prob-ability of 20%). The same function would have a different criticality/probability score (see below), based on the fact that two different failures could hit that business function.
The business criticality evaluation process is strictly related to the risk management process and is useful for any project stakeholder, at any project stage.
Measuring Business Criticality
To be successful, any test risk assessment has to concentrate on the local identifiable issues related to the business. During the test risk assessment process, risks to business functions will be identified and evaluated. The vulnerabil-ity of the business to these risks will be rated. Main actions to perform in a test risk assessment include:
• Alignment of testing with business requirements.
• Test strategy and test environment preparation.
• Test resourcing.
• Test environment.
• Test execution.
• Production readiness preparation.
Project Costs Increase at Testing Stage
Late Defect Discovery Results in Significant ReworkDefect
PreventionNew
Rat
e of
Cos
t In
vest
men
t D
urin
g P
roje
cted
Life
cycl
e
Requirements Design and Build Release to Test
Release to Field
Time
100x Increase in Cost of Removing Defects
Sources: Barry Boehm, “Software Engineering Economics,” Prentice Hall, Inc., 1981. Basili Boehm, “Software Management,” IEEE Computer, January 2001.Figure 1
cognizant 20-20 insights 3
Each of these business functions represents a dimension to be analyzed in setting up a risk management process.
Profiling Risk Level of Projects
After having defined main dimensions and sub-dimensions as a core part of the TeRAPI method-ology, each project must be assessed to weigh the risk associated with each sub-dimension in each stage of the project lifecycle.
A business criticality score is determined based on the business purpose of the project. The result is a business risk index (BRI), which offers a composite view and comparison of risks based on the business criticality of the projects that require immediate management attention, as well as the priorities of these projects.
All dimensions are quantitatively determined, from business requirements alignment, to test result analysis and mitigation plan design — and a qualitative index is assigned. Frequency of risk across each action step must be determined and graphed.
For each dimension and its specific sub-dimen-sions, a qualitative measure of risk importance needs to be identified and quantified. This must be done across all projects where TeRAPI is applied.
This data is placed in a qualitative table that summarizes all quantified risk-level results for
each assessed project in an effective way and in relation to each dimension and relative sub-dimensions (see Figure 2).
The TeRAPI risk assessment identifies two types of risks for testing:
• High-level risks: Risks that are likely to impact the schedule or the quality of the deliverable. The root causes for these risks need to be identified and addressed through initiatives at the higher level.
• Medium-level risks: Risks that can be addressed by the testing team with minimal impact on schedule or quality. These risks can be addressed by project managers through tactical initiatives.
Once the test risk assessment is completed, the next step is to analyze and present the results so that executive management can get the most use from the data. Data analysis will be the foundation for planning actions to mitigate risk impact and provide recommendations.
The recovery strategies that need to be developed should be based on the findings of the risk assessment survey and interviews, as well as the business impact analysis.
Designing the Risk Mitigation Plan
Each TeRAPI dimension must be developed and risk causes determined and analyzed to create
TeRAPI: Project Test Risk Evaluation
TeRAPI DIMENSIONS PROJECTS
Dimensions #1 #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15Sub-dimension #1 H M H H H M M M M
Sub-dimension #2 H H H M H M M
Sub-dimension #3 H H M H
Sub-dimension #4 H M M M M M
Sub-dimension #5 M M M M M M
Sub-dimension #6 M M
Sub-dimension #1
Sub-dimension #2
Sub-dimension #3
Sub-dimension #4
Sub-dimension #5
Sub-dimension #6
High Medium
High (H)Risks that are likely to impact the schedule or quality of the deliverable Medium (M)Risks that can be addressed by the project manager with minimal impact on schedule and quality
0 2 4 6 8 10
0 2 4 6 8 10
4
3 3
3 1
1
5
5
6
2
Figure 2
Frequency of Risk for Individual Projects
Projects
0 5 10 15 20
2456
2620
2566
2761
2281
2881
2681
2847
2673
2710
2776c
2776b
2815
2912
2817
15 4
11 6
7 10
6 9
9 2
2 8
1 8
4 5
2
2
6
7
1 5
1
1
1
4
4
cognizant 20-20 insights 4
a mitigation plan that addresses them. The mitigation plan, or risk response plan, provides options to reduce the likelihood that identified risks will occur.
Options and actions are developed in a mitigation plan to enhance project opportunities and reduce threats to project objectives to “below an acceptable threshold.”
Typically, every testing team maintains a mitigation plan, which consists of an Excel file containing different columns, including risk description, date, category, impact areas, root cause, priority, contingency, action plan, mitigation action plan, risk status and risk close date.
Each mitigation action is related to the corre-sponding identified risk. Each action must be planned and time-lined in order to track it and update the probability that the corresponding risk will occur.
The mitigation plan is managed by a supervisor team belonging to the testing organization and responsible for developing mitigation actions that will address all identified risks. To facilitate workload assignments, various workstreams can be created and assigned to different execution teams.
If ranked risks are not mitigated properly, they will occur at certain points in time during testing activities. In that case, the risks become issues, and the supervisor team is responsible for iden-tifying viable countermeasures and action plans to resolve them.
TeRAPI Process Implementation: Business CaseA continuous risk management process is a necessary part of any approach to software security. In particular, at the testing stage, the primary objective is to improve the understand-ing of the major risks, in order to ensure stable and reliable testing. The TeRAPI approach can help test managers increase confidence in the tests of security function behaviors.
Risk assessment is widely applied in all market industries and large organizations, but it is partic-ularly important in banking and financial services. “Run-the-bank” projects implement a well-struc-tured risk assessment process in order to manage all approved change requests by minimizing the risks during code testing. This process is set up at the scope identification stage.
The supervisor team, which includes the sponsor, project manager, solution lead engineer, require-ments engineer and test manager, is responsi-ble for gathering the requests coming from the business and setting up an implementation plan in order to get them deployed according to the project plan. During the evaluation of the project scope, the team sets up a risk plan to evaluate potential constraints during solution implemen-tation that would potentially occur during the requirements definition, solution development and code testing stages.
For a tier one financial institution in Europe, two run-the-bank projects are considered, one without a risk management application, and the other with the TeRAPI application (see Figures 3 and 4). The project scope is to implement several change requests (CRs) for online private banking services. Both projects have a similar normalized initial budget estimated at CHF 400,000.
The implementation of a structured risk assess-ment process, based on TeRAPI principles, has produced a cost savings of 19%, or around CHF 76,000, and reduced identified defects from 17 to 10 (about 60% less).
Before and After TeRAPI
Project 1 % Difference
Initial budget CHF 400,000
Number of defects at testing stage
17
Additional budget allocation to fix testing bugs
CHF 55,000 13.8%
Cost savings – CHF 55,000 -13.8%
Project 2 % Difference
Initial budget CHF 400,000
Number of defects at testing stage
10 -58.8%
Additional budget allocation to fix testing bugs
CHF 0
Cost savings CHF 20,000 5.0%
Net cost savings CHF 75,000 18.8%
Figure 3
cognizant 20-20 insights 5
Project 1 has managed about 12 CRs without a risk management plan. It experienced 17 issues at the unit test stage, mainly related to not meeting requirements in code development. This resulted in a large amount of rework and the need for additional resources to cope with unexpected issues, increasing the project budget to CHF 55,000.
In Project 2, the sponsor decided to use TeRAPI principles in an effort to reduce project expendi-tures, particularly during the testing stage. The supervisor team was established, and risks were identified. At the beginning, the risk was very large, with 50 identified risks. For each risk, a set of mitigation actions were established, with
a proper due date and responsibility assigned to members of the team.
Prior to unit testing, the number of unaddressed risks was reduced to 10. During testing for the first release of the year, the number of issues to be addressed dropped by 60%, producing savings of CHF 20,000 compared with Project 1. The net savings obtained by Project 2 was about CHF 75,000.
All identified risks were properly weighted to define risk relevance. The impact on the project implementation strategy, schedule, functionality, costs and quality were weighed, and each value concurred to define the risk probability.
TeRAPI Implementation: Project Advantages
-60
-40
-20
0
20
40
60
80
Additional budget allocation
Cost savings Net cost savings
Project 1
Project 2
Number of defects at testing stage
Figure 4
TeRAPI: Risk Assessment Landscape
Top 15 Risks
Risk Type
Risk Category
Project Risk
IT Risk
Operational Risk
Project Management
Product Development
Sponsorship and Budget
External Risks
Risk Type (Open Risks)
Project Risk 3IT Risk 1Operational Risk 1
Risk Category (Open Risks)
Project Management 1Product Development 2Product (procured, operational) 1
Sponsorship and Budget 1External Risks 0
Low MiddleImpact
High
Pro
babi
lity
0
25
50
75
100
20%
20%
20%
20%
20%
40%
60%
Figure 5
cognizant 20-20 insights 6
The higher ranked risks were shown in a context that summarized all major risk properties, such as the probability to occur, risk type and category.
Each risk was given a set of mitigation actions, with a relative supervisor team member assigned as risk owner and responsible for putting into practice all identified mitigation actions and tracking risk addressing. A risk assessment statement form (see Figure 6) collects all infor-mation concerning identified risk and is the kernel source for the final report.
TeRAPI Final ReportThe risk landscape and mitigation plan will show a clear overview of risks, root causes and action agenda to resolve all challenges. The findings from TeRAPI will form the basis for the final report. A risk assessment outcome is the starting point for building the TeRAPI final report. The risk landscape shows the status of major risks to be tracked and reviewed periodically, trends across different reviews and the main categories to which the risks belong.
The final report will take the following into con-sideration:
• Previous disruption history.
• Risks and vulnerabilities.
• Preventative measures.
• Test risk assessment.
• Mitigation plan.
The risk assessment process is an essential activity of continuity planning, in particular during the testing phase. Catching errors prior to testing the developed solution helps save time, money and effort and will result in cleaner, better performing and more business-aligned code.
The TeRAPI process can enable agile and effective IT systems to support a more effective business. The purpose of the TeRAPI methodology during software testing is to identify high-risk appli-cations that must be tested more thoroughly, as well as identifying error-prone components within specific applications that must be tested
TeRAPI: Risk Assessment Statement
Figure 6
Impact on Project
30 3 (1- 9) 7 (1- 9) 7 (1- 9) 5 (1- 9) 3 (1- 9)
Response Planning:
Response Action Cost Responsibility Type Due Date
11'500 Mitigation G
2 A500 MitigationStatus:3 1'000 Mitigation A4 1000 Mitigation A
Project Management
Risk Owner: Risk Grade 3 Risk GradeLast updated by: (Risk)
Project: Program:Last update on:
05-Mar-12Project ManagerPre-release testing to increase test coverage.
Costs: 10'000
Risk Exposure: 3000.0027-Mar-12Project Manager 2.10 0.90Project Manager
Risk Type: IT Risk Project Area:Risk Category: Product Development
Reason for Closure
Closed on:
Regular followup with release management team and timely escalations. Lead Engineer 05-Mar-12Open
Close coordination and weekly tracking with testing team and external support to verify and address identified test risks. Test Manager 05-Mar-12Unit test execution planning reviewed daily with external resources responsible. Project and Test Managers 05-Mar-12
Agreed functionalities during scope definition could not be delivered 100%
Schedule Functionality Costs Quality
Response Effectiveness
Probability Impact
on ProgramRisk (%) Strategy
Identification Date: 07-Jan-12 Test Manager
49
Description: Due to the project’s dependencies on external parties, there’s a threat that release CRs’testing could be delayed, which will result into overall timeline shift
Cause/Origin : Project organizational structure involving multiple external parties
Consequences: Overall delay on release timeline
Risk ID Keyword: (Release Aug’12) - Unit tests execution warning
cognizant 20-20 insights 7
more rigorously. The results of the analysis can be used to determine the testing objectives for the application during test planning.
ConclusionCompanies have always had to prioritize their testing efforts, but figuring out how to do it properly — with the right approach and method-ology — has not always been clear. To compete, enterprises must reduce testing costs as low as possible without sacrificing application quality or delaying critical application launch time.
TeRAPI can help in this regard. It is backed by statistical techniques and proven best practices gleaned from many years of experience with testing code of enterprises worldwide. Our approach ensures the appropriate amount of input from business and technical experts, as well
as the proper level of detail in prioritizing risks, at each stage of the project lifecycle.
In order to implement this assessment process effectively, we estimate a team of three or four experts is needed to implement the process and prepare it for practical application for all involved projects.
Our TeRAPI methodology is an effective way of minimizing risks during the testing stage, with the objective of minimizing delays and costs, and ensuring the highest business benefits. This approach provides companies with a consistent, business-based methodology for determining the level of risk present at the testing stage and how to mitigate it in order to deliver business value and the highest possible customer satisfaction.
About the AuthorsSimon Erdmann is a Director at Cognizant Business Consulting (CBC) in Frankfurt, Germany. He has over 15 years of experience in operations, management and consulting for the retail, consumer goods and transportation logistics industry, and is head of Germany, Austria and Switzerland for CBC Strategic Services. Simon has studied at German Armed Forces University and VWA Essen. He can be reached at [email protected].
Giuseppe L. Pannisco is a Consultant at Cognizant Business Consulting. He has six-plus years of consulting experience with global clients on supply chain management, IT strategy and management issues, working with companies in the automotive, aviation and aerospace, transportation and logistics service provider fields. Giuseppe holds a master’s degree in aerospace engineering (aeronautical business management specialization) from Universita’ degli Studi di Napoli “FEDERICO II.” He can be reached at [email protected].
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out-sourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 50 delivery centers worldwide and approximately 140,500 employees as of March 31, 2012, Cognizant is a member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters500 Frank W. Burr Blvd.Teaneck, NJ 07666 USAPhone: +1 201 801 0233Fax: +1 201 801 0243Toll Free: +1 888 937 3277Email: [email protected]
European Headquarters1 Kingdom StreetPaddington CentralLondon W2 6BDPhone: +44 (0) 20 7297 7600Fax: +44 (0) 20 7121 0102Email: [email protected]
India Operations Headquarters#5/535, Old Mahabalipuram RoadOkkiyam Pettai, ThoraipakkamChennai, 600 096 IndiaPhone: +91 (0) 44 4209 6000Fax: +91 (0) 44 4209 6060Email: [email protected]
© Copyright 2012, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
