Embed Size (px)
Citation preview
Jacek Winiarski*
Comparative analysis
of risk assessment methods in project IT
Introduction IT projects implementation as described in software development
methodologies is usually distributed into several stages. It begins with
requirements specifications and ends with operation and maintenance of
the implemented product [Chapman, Ward, 1997, p. 168]. Throughout the
course of software development cycle the events affecting its course may
take place. In practice, they are described by probability of their
occurrence and the potential scope of damage they may create in the
implemented project. The product of these quantities is defined as the risk
measure [Frączkowski, 2003, p. 128]. It is used for forecasting negative
impacts on the course of the implemented individual activities within the
task or the entire project. Project management is the approach oriented at
the accomplishment of the set objectives within the assumed time and
budget [Kaczmarek, 2005, p. 65]. It is an art of maintaining the project
failure risk at a possible lowest level throughout the entire project cycle.
Risk management is one of many elements of the project management
process. In principle, it is distributed into the following stages [Chong,
Brown, 2001, p. 45]: identification and distribution of risk sources,
identification of exposed project tasks, risk assessment, planning of
response to the risk and risk and risk monitoring in the course of project
implementation.
1. Failures in implementation of IT projects There are scores of institutions acting in the field of risk analysis in
IT projects. These are mainly academic centres but there are also many
organizations associating experts with know-how. One of the most often
cited publications are reports called The CHAOS Chronicles, published
regularly by The Standish Group International, an American institution
dealing with monitoring of IT projects implemented in the USA. The
* PhD, Department of Electronical Business, Faculty of Economic, University of Gdansk,
email: [email protected]
Jacek Winiarski 180
analysis of documents published on the Web fosters detailed specification
of the results of IT projects monitoring in the form of statistics.
According to the data publish on The Standish Group International
web pages, approximately 2/3 of IT projects fail to end in a full success.
Throughout their implementation (according to the statistics in every
second project), there are deviations from time schedule or budget
assumptions, failure to develop all the designed functions of the
programme or abandonment of the entire project. The main objective of
the risk assessment at the planning and implementation stages of IT
projects is to reduce the number of projects which are likely to exceed the
scheduled resources or be eventually terminated [Szyjewski, 2004, p. 125].
The risk assessment techniques presented in the paper compel the IT
Project Managers to continuously update the information on potential
hazards. One of the frequent mistakes made by IT Project Managers is to
assume that the risk of successful accomplishment of the project remains
the same throughout the entire project of its implementation [Pańkowska,
2001, p. 84]. It is to the contrary. The risk is incessantly variable, thus
constant monitoring as per a carefully designed schedule is of essence. As
shown in practice, the meticulousness and thoroughness of the risk
assessment contribute considerably to the end success of the project.
2. Characteristics of the selected IT project The study was based on a project of system development aimed at
servicing the Lending Library of Higher Education School. The
undertaking in question was designed as a set of 349 tasks to be carried
out by a group of 5 IT Specialists within 64 days. The accomplished and
implemented software is to facilitate students’ access to the book
catalogue by means of web browsers.
The detailed comparative analyses were based on eight tasks from
the entire project exposed to the risk of failure, taking into account the
scheduled time, scope and budget (Table 1).
Table 1. Selected risk-prone projects tasks
ID Task name: Duration
Probability of non-
compliance with the
scheduled resources
Implementation
costs [PLN]
40 Requirements
Specification 14 days 0,4 4500
Comparative analysis of risk assessment methods in project IT… 181
ID Task name: Duration
Probability of non-
compliance with the
scheduled resources
Implementation
costs [PLN]
43 Definition of
classes 3 days 0,3 800
52 Code
development 30 days 0,5 10 300
67 Code testing 5 days 0,6 4 400
104 Code adjustment 10 days 0,5 7 300
109 Software
installation 30
minutes 0,3 500
110
Preparation of
user
documentation
10 days 0,3 5 500
111 User training 2 days 0,2 4 800 Source: Own elaboration.
3. Examples of application of risk assessment techniques in the
selected IT Project – case study
The popular risk assessment methods applied to IT Projects
comprise: 2x2 matrix method, probability and effects matrix method,
Heeg’s method, failure Analysis of failure effects. Less popular
techniques encompass: sensitivity analysis, spot techniques, probability
analysis (using e.g., Monte Carlo simulation), flow diagrams (e.g., critical
path analysis) or decision tree analysis (e.g., PERT, VERT, GERT
analyses).
3.1. 2x2 Matrix Method
One of the primary tools fostering risk management process in
project works is the so-called 2x2 matrix. It defines risk as the probability
function for the occurrence of harmful event and the effect thereof
[Pritchard, 2002, p. 122].
Table 2. Risk assessment using 2x2 Matrix method
Impact
Probability Small Large
Large Quarter 1
40’
Quarter 2
40
Small Quarter 3
40’’
Quarter 4
Source: Own elaboration.
Jacek Winiarski 182
2x2 Matrix should be completed following the previous
identification of potential hazards and preparation of the list thereof.
Next, they are filled (depending on the probability of occurrence and the
scope of potential loss they may cause) in their respective matrix quarters.
The parts of the table specify [Chong, Brown, p. 65]:
– Quarter 1 represents the area of hazards of high probability of
occurrence and inconsiderable negative effects for the project,
– Quarter 2 represents the area of hazards of high probability of
occurrence and simultaneously substantial negative effects for the
project implementation process,
– Quarter 3 represents the area of hazards of small probability of
occurrence and inconsiderable negative effects for the project
implementation process. This is the least risk-prone area.
– Quarter 4 represents the area of hazards of small probability of
occurrence and considerable negative effects for the project,
If the 2x2 table is not completed with hazards but with the tasks
exposed to the risk of non-compliance with the scheduled resources, then
after the analysis of the possibility of reallocation thereof along the
directions recommended by the technique in question, precautions will
be developed with the view of diminishing the risk for the
accomplishment of particular tasks and thus the entire project.
Task 40 – requirements specification was placed in the 2nd quarter
by the project risk manager. Next, the Project Manger suggested that Task
40 should be performed in compliance with the forms adopted in
PRINCE2 method. The suggestion caused the task to be moved to the first
quarter 40’ (prim variant). Next, another Project Manager decided to carry
out additional audit of the prepared requirements specification by an
external expert. This decision caused the task 40’’ to be moved to the third
quarter 40’ (bis variant). All project tasks may be examined individually
in the same manner.
3.2. Method of Matrix of probability and effects
Another more complex tool is the so-called matrix of probability and
effects. It is an elaboration of the 2x2 matrix concept. It is more detailed as
far as the probability of estimates and the effects of hazard occurrence are
concerned. Like previously, particular hazards are filled in the respective
fields in the extended version of the table. After all hazards identified in
the project have been filled, preventive measures are designed to
eliminate the risk sources allocated in the second quarter [Chong, Brown,
Comparative analysis of risk assessment methods in project IT… 183
2001, p. 64]. The ultimate objective of the risk management process is the
reallocation of the most probable and most perilous hazards tops other
areas of the matrix. The measures provide the basis for the risk
management in the project planning process.
The table used for the method in question may upon its modification
serve for calculation of the measurable total scope of project risk. The
aforesaid modification provides for allocation to particular cells of hazard
weights representing the scope of probability of occurrence of a given
hazard and the potential effects thereof (Table 3).
Table 3. Risk Assessment Method of Matrix of Probability and Effects
Probability Effects
Minimal Minimal Minimal Minimal Minimal
Extremely high
(0.8 – 1)
43
(2,0)
40
(3,5)
(7,0)
67
(8,0)
(9,0)
High (0.6 – 0.8)
(1,5)
52’
(2,0)
52
(5,0)
(7,0)
(8,0)
Average
(0.4 – 0.6)
43’
(1,2)
67’
(1,8)
104
(4,0)
(5,0)
(7,0)
Low (0.2 – 0.4) 40’, 104’
(1,0)
110, 109
(1,5)
(3,0)
(4,0)
(5,0)
Extremely low
(0 – 0.2)
110’,109’, 111’
(0,5)
111
(1,0)
(1,5)
(3,0)
(4,0)
Source: Own elaboration.
Each identified hazard, which may occur during performance of the
project, shall be allocated to particular cells of the table. As a next step, the
weight of a given cell should be multiplied by the number of hazards
allocated thereto and sum up all the achieved numbers. The sum shall be
divided by the total number of hazards in the analysed project. The end
result is a measurable quantity of the total project risk.
With respect to the project, the entire introductory risk of the project
was 3.31. After the preventive measures have been recommended and
applied, the total project-related risk was reduced to 1. The presented
example, like 2x2 matrix method was based on the study of risk-prone
tasks and not the tasks themselves.
Jacek Winiarski 184
3.3. Heeg’s method
The method recommended by Heeg in [Chong, Brown, 2001, p. 167]
comprises three stages. These include:
– risk identification,
– risk assessment,
– selection.
The presented method is based on the identification of project-
specific hazards. According to the author of the method, the risk sources
may be the identified in several ways. One of the commonly used
techniques is the analysis of task packages described by means of e.g.,
Work Breakdown Structure - WBS. It may be presented in the form of a
Table 4.
Table 4. Risk assessment using Heeg’s method
ID Task name: Potential
risks
Probability
of occurrence
Costs
of
neutralization
[PLN]
Probable
costs [PLN]
40 Requirements
Specification
Omission
of
required
functiona
lities
0,4 8000 3200
43 Definition of
classes
Incomple
te classes 0,2 1000 200
52 Code
development
Syntactic
errors 0,01 500 5
67 Code testing
Data
transfer
errors
0,3 4000 1200
104 Code
adjustment
Semantic
errors 0,04 300 12
109 Software
installation
Incompat
ibility 0,1 2000 200
110
Preparation of
user
documentation
Deadline 0,01 3000 30
111 User training Deadline 0,01 1500 15
Source: Own elaboration.
Comparative analysis of risk assessment methods in project IT… 185
Following identification of the risk-prone tasks and detailed
specification of potential risk sources, which may affect the
implementation process, it is necessary to determine the probability of
occurrence of detailed hazards (Table 4, Column 4). Next, the planned
costs related to elimination of potential losses are to be estimated (Table
4, Column 5). The last column of Table 4 comprises probable costs i.e.
product of probability and foreseen costs of loss compensation (Table 4,
Columns 4 and 5).
Thus computed quantities of probable costs must be sorted in
descending order and the group of tasks for which the sum of quantities
in Column 4 Table 4 will be 75% of the total probable costs of the analysed
project [Chong, Brown, 2001, p. 98] must be specified (starting from the
highest values). In the example in question, these include tasks 40 and 67
(amounting to 90.5% of the total costs). Thus identified set of task groups
shall be given a particular attention from the Project Managers. The
possibility to undertake protective measures for these groups must be
taken into consideration. The sum of total probable costs shall be 4862
PLN.
3.4. Failure Mode Effect Analysis
Failure mode effect analysis was proposed by Maylor and described
in [Chapman, Ward, 1997, p. 87]. This method analyses three parameters
describing all tasks within the project. Each of these parameters must be
expresses as a number on a scale from 0 to 10. The author adopts one point
scale for all parameters. The requested quantities include:
– meaning of failure of implementation of a given task (failure),
– probability failure oversight,
– probability of failure occurrence during performance of a particular
task.
Each of the parameters must be examined individually. The objective
of the presented analysis is to calculate a given total risk task constituting
a function dependant on the aforementioned parameters. The risk is
calculated on the basis of the following dependence:
Risk = failure significance * probability of failure omission
* probability of failure occurrence
Jacek Winiarski 186
The higher the risk values, the more serious hazard is related to a
particular task. With respect to activities exposed to the highest risk,
additional measures alleviating potential losses should be proposed.
Table 5. Risk Assessment Through Failure Effects Analysis
ID Task name: Failure
significance
Probability
of failure
oversight
Probability of
failure
occurrence
Risk
40 Requirements
Specification 8 3 4 96
43 Definition of
classes 3 2 3 18
52 Code
development 7 1 5 35
67 Code testing 8 2 6 96
104 Code
adjustment 7 2 5 70
109 Software
installation 5 1 3 15
110
Preparation of
user
documentation
4 7 3 83
111 User training 5 1 2 10
Source: Own elaboration.
For each task examined in the project by Failure Mode Effect
Analysis, two additional parameters must be provided. These include:
failure significance and failure oversight probability. Having performed
the calculations illustrated in Table 5, one may discern that the highest
risk pertains to tasks 40 and 67, while the least to the tasks no. 109 and 18.
The total project risk is the sum of values in the last column, which is 423.
4. Comparative analysis of applications of risk measurement
methods in IT projects Applications of the results obtained by means of 2x2 Matrix Method
are not vast. This method is suitable for presentation of risk mitigation
issues, since it clearly illustrates the required trends of preventive
measures. IT project risk assessment based on this method may occur
vague, general and eventually not yielding satisfactory results for the
Managers. The concept of 2x2 matrix is focusing on risks, not the risk-
Comparative analysis of risk assessment methods in project IT… 187
prone tasks, which definitely affects the profile of the analyses being
carried out. 2x2 matrix does not allow risk quantification, with respect
either to a part or the entire project. This technique is an easy-to-use tool
for risk assessment in small projects. Its application supports the strategy
of compensating potential effects of identified risks.
If 2x2 matrix is completed with risk-prone tasks instead of identified
risks, then after the main assumptions of the method have been applied,
the effect of its use shall consist in development of preventive measures
plan targeted at reduction of risks for individual tasks, hence for the entire
project. Thus applied 2x2 matrix method will facilitate the assessment of
the proposed preventive measures (after twice risk quantity
measurement, before and after the preventive measures have been taken).
Those who wish to use this technique shall be good experts in risk
management, as this technique is based on the intuition, which
determinant for the usability of the obtained results.
Probability and effects matrix is the extended version of 2x2 matrix
method. It has two advantages, which differentiate it significantly from
the original pattern. One advantage is the fact that it fosters calculation of
risk for individual tasks, task groups or the entire project, before and after
the preventive measures have been taken. The other advantage is the
clarity of results, not only for small but also for medium-sized IT projects.
Heeg’s method is the first from among the presented techniques,
which assigns individual risks to the planned project tasks. Each of the
tasks may be assigned more than one risk. This technique requires
specification of the probability of occurrence of all identified risks. It is an
interesting parameter, since within the framework of the risk definition
another quantity is searched for, which is the probability of task non-
performance [Knight, 1934, p. 120]. In Heeg’s method it is indispensable
to specify potential costs of the reduction of effects of the identified risk
occurrence. In practice, both quantities are identified on the basis of
experience and intuition of the researchers. The method facilitates a
detailed identification and analysis of risk sources, which may occur
during the implementation of the project. By means of these method, the
sources may be easily identified and assigned to particular tasks. This
possibility is an essential advantage of the method in question.
Unfortunately, this method also employs heuristic quantities in final risk
assessment.
Jacek Winiarski 188
In failure mode effects analysis it is necessary to specify further
parameters. These parameters are not required in any other risk
assessment techniques. It is indispensable to determine: failure
significance, failure oversight probability as well as failure occurrence
probability. The last parameter is identical to the scope of risk of a non-
performed task taking into consideration scheduled time resources, scope
and budget. The person managing the risk must express all these values
on a scale from 0 to 10.
Failure mode effects analysis does not employ mathematical tools
facilitating objectivity of data use for calculations. Like the
aforementioned methods, the values used are based on the intuition. It
will prove, however, in comparative analyses. From among the presented
methods, this one allows the largest number of details to be used in the
study. It occurs that Managers value the possibility of taking into account
the probability of failure oversight. Failure mode effects analysis may be
easily used for the risk assessment in large and middle-sized IT projects.
Table 6. Comparative applications of risk assessment applications IT projects
Method name
Required resources Results
Co
sts
Du
rati
on
of
imp
lem
enta
tio
n
Eas
ines
s o
f ap
pli
cati
on
Tim
e in
vo
lvem
ent
Pre
cisi
on
Usa
bil
ity
2x2 Matrix Method l s e s l l Method of Matrix of Probability and Effects l s e m m m Heeg’s Method l s m h m h
Failure Mode Effect Analysis l s d m h h
Key: low (l), medium (m), high (h), short (s), easy (e), difficult (d).
Source: Own elaboration.
Conclusion The paper depicted four methods of risk assessment methods
applied in IT projects implementation. The same part of the IT project was
examined through a comparative analysis. The obtained results foster
concept that matrix techniques focus predominantly on the analysis of
identified risks. They only indirectly examine the project tasks, for which
it is probable, not comply with the scheduled time resources, scope or
Comparative analysis of risk assessment methods in project IT… 189
budget. As a consequence, although these methods mark out the
directions of preventive measures aimed at reduction of the risk, they
seem not to be useful for large or complex IT projects. They are suitable
for rough analyses in small projects.
Further two methods foster a more detailed risk assessment. Based
on the identified potential risks assigned to particular project tasks,
Heeg’s Method specifies probable costs, the company will have to incur
in case of an anticipated risk. This project does not foster total risk
calculation. This technique is suitable for comparative analyses for several
variants of implementation of the same task, while it is not appropriate
for the entire IT projects.
The last presented technique – Failure Mode Effect Analysis
introduces two additional (crucial) parameters: failure significance and
failure oversight probability. Thanks to these parameters, the method
enables calculation of risk for implementation of all individual tasks as
well as the entire project. This technique is commonly used in IT project
risk management as the calculations involving two new parameters are
very useful.
The Project Manager’s ultimate decision on the choice of the method
for risk assessment in planning and implementation of an IT project will
depends first and foremost on the specificity of project requirements
(scope and innovativeness), funds and selected implementation methods.
References
1. Chapman Ch., Ward S. (1997), Project risk management processes,
techniques and insights, J. Wiley & Sons, Chichester.
2. Chong Y.Y., Brown M.E. (2001), Zarządzanie ryzykiem projektu,
Oficyna Ekonomiczna, Dom wydawniczy ABC, Kraków.
3. Frączkowski K. (2003), Zarządzanie projektem informatycznym, Oficyna
Wydawnicza Politechniki Wrocławskiej, Wrocław.
4. Kaczmarek T.T. (2005), Ryzyko i zarządzanie ryzykiem. Ujęcie
interdyscyplinarne, Difin, Warszawa.
5. Knight F. (1933), Risk, uncertainty and profit, London.
6. Pańkowska M. (2001), Zarządzanie zasobami informatycznymi, Difin,
Warszawa.
7. Pritchard C.L. (2002), Zarządzanie ryzykiem w projektach. Teoria i
praktyka, WIG-PRESS, Warszawa.
Jacek Winiarski 190
8. Stabryła A. (2006), Zarządzanie projektami ekonomicznymi i
organizacyjnymi, Wydawnictwo Naukowe PWN, Warszawa.
9. Szyjewski Z. (2004), Metodyki zarządzania projektami informatycznymi,
Wydawnictwo PLACET, Warszawa.
10. Winiarski J. (2007), Analiza metod zarządzania ryzykiem w pracach
projektowych z dziedziny informatyki, Pieniądze i Więź, Nr 2 (35),
Gdańsk.
Comparative analysis of risk assessment methods in project IT
(Summary) The paper is targeted at the comparative analysis of practical applications
of risk assessment methods of projects in IT industry. The study was based on an
IT Project comprising 349 activities, wherein 8 were selected which were the most
likely not to comply with the scheduled time, scope and budget resources. Next,
four techniques were described and apply to provide the assessment of the same
part of the project: 2x2 Matrix Method, Probability and Effects Matrix Method,
Heeg’s Method, Failure Mode Effects Analysis. The obtained results were
discussed and on the basis thereof, conclusion were formulated to select the
method depending on the specificity of an IT Project.
Keywords IT projects, risk management, risk assessment techniques.
