Viewpoint-based Risk Assessment and Prioritization

Hareton K.N. LeungDepartment of Computing

The Hong Kong Polytechnic University Hong Kong

cshleung@inet.polyu.edu.hk

Abstract – For software projects, different stakeholders may place different emphasis on the same risk. Risks that are important to one stakeholder may be less important or irrelevant to other stakeholders and vice versa. In the overall prioritization of risks for a software project, we need to consider this viewpoint-based difference. This paper first introduces the concept of viewpoint-based assessment of risks and then presents three methods for generating the overall risk prioritization based on the separated viewpoint-based prioritizations. We also present a case study of an outsourced project by applying the proposed methods and illustrate their relative strengths and weaknesses.

Keywords–Viewpoint-based, Risk assessment, Risk prioritization

I. INTRODUCTION

A key step in risk management is identification of risks [1,2,3]. As most organizations have limited resources devote to managing risks, they do not normally deal with all the identified risks. A common approach is to prioritize the risks and focus on the “top” risks. For example, Boehm advocated managing just the top ten risks [4].

There has been little guidance in selecting the “top” or high priority risks. For example, some organization may use the following prioritization criteria or some combination of them:

� The risk is important according to senior management � The risk is important according to the project manager � The risk has a high probability of occurrence � The risk may occur earlier than others. The notion of viewpoints has been used in requirements

engineering for a long time [5-8]. The discrepancies between different stakeholders can be used to improve the requirement elicitation and other aspects of software development. We borrow this notion of viewpoints for risk identification and prioritization.

Risk prioritization is a complex process that often involves subjective judgement and requires many discussion and negotiation among stakeholders. In general, a risk prioritization order is highly dependent on the stakeholders’ viewpoints. For example, a project manager may place an on-time delivery risk higher than that of a technical risk, while a software designer may put higher emphasis on the technical risk. However, current practice of risk prioritization does not explicitly consider the separated priority assigned by various stakeholders. Since there are many different stakeholder

groups (such as customer, project team members, etc), before taking any mitigation actions, all stakeholder groups should come up with a consensus on the prioritization of risks.

A common method for risk prioritization is the weighted score method, where each factor is assigned a score that defines its relative level of risk [15]. Then, the risk is multiplied by the relative importance of risk factor among all factors. Recently, researchers have proposed a fuzzy approach combined with a group decision-making technique [16]. In particular, fuzzy entropy is utilized with multiple criteria group decision making to prioritize the risks.

In this paper, we first propose a viewpoint-based approach to identify risks. In a certain sense, we are re-classifying the risk types by different viewpoints. Then, we present three methods for generating the overall prioritization of risks, given the risk prioritization of each viewpoint.

The contributions of our study include: (1) propose a viewpoint-based risk assessment, (2) design three methods for integrating risk prioritizations by viewpoints into an overall risk prioritization, and (3) illustrate the applicability of our proposed prioritization methods with a case study.

The paper is organized as follows. Section II introduces the concept of viewpoint-based assessment of risks. Section III proposed three methods for generating an overall risk prioritization based on the separated viewpoint-based prioritizations. Section IV presents a case study of an outsourced project by applying the proposed prioritization methods and our key observations. Section V offers our conclusion and an outline of future research.

II. VIEWPOINT-BASED RISK ASSESSMENT

Different project stakeholders have different goals and concerns about the software project. For example, the customer may be more interested in getting the project done at the lowest possible cost, while the supplier may aim for as high profit from the project as possible. When presented with a set of risks, some stakeholder may prioritize certain risks higher than others, while another stakeholder may come up with a very different risk prioritization order. To better model this behavior, we first introduce a set of viewpoints to represent different groups of stakeholders. These viewpoints are derived partly from the key responsibilities of stakeholders and their project goals.

In the context of software project management, we can identify two key parties, namely, client (or buyer)

2012 IEEE 35th Software Engineering Workshop

1550-6215/13 $26.00 © 2013 IEEE

DOI 10.1109/SEW.2012.27

193

organization and supplier (or seller) organization. These two parties can be easily identified for outsourced projects. For internal project, we can identify the serving team as the supplier and the project owner as the client. For internal IT development project, the supplier is often the development organization. Within the supplier organization, we have the following key viewpoints (or roles):

Supplier senior management – This role represents the interest of the supplier organization. The key interests are successful business, customer satisfaction, and profit from the project.

Project manager – This role ensures the project is successful in term of the defined success criteria. Anything that may affect the project success is of concern, such as project completion time, cost and quality.

Project team member – This role ensures the assigned task, such as requirements definition, design, code and testing is completed satisfactory.

Within the client organization, we can identify two key viewpoints:

Customer – This role represents the client and mainly interests in the deliverables of the project. Key interests include on-time delivery, on-budget delivery, complete scope, high quality, and achievement of project objectives, such as cost saving, improve sale, etc.

User – This role uses the deliverables of the project. The key interest includes system quality attributes like user friendliness, functionalities, safety, etc.

Table 1 summarizes the viewpoints and their key focus. The viewpoints are directly related to the stakeholders. Each group of stakeholder is unique and has its own set of goals. These goals determine the priority of the project (and also the priority of risks). We assume that stakeholders within each group will work together to identify the risks and also their priority. It is essential that each stakeholder group arrives at a commonly acceptable prioritization of risk before negotiating with other groups on the final prioritization of the risks. Thus, discussion and negotiation occur within each group first and subjective judgement (on probability and impact) is taken into consideration. In fact, within each group, they can also apply some form of viewpoint based prioritization.

As the goals of each group of stakeholders may overlap, the priority of risks may also overlap among the various groups of stakeholder (or viewpoint). Figure 1 shows the relationship between viewpoints, stakeholders, project goals and priorities.

TABLE 1: VIEWPOINTS AND THEIR KEY FOCUS

Organization Viewpoint of stakeholder group

Focus

Client Customer Client organizational focus User System functionality, usage Supplier Senior management Supplier organizational focus Project manager Project focus Project team member Work or Task focus

Figure 1: Relationship between Viewpoint, Stakeholder, Project goal and

Priority Different viewpoints place different emphasis on various

risks. Table 2 lists some example important risks according to various viewpoints.

TABLE 2: EXAMPLE RISKS BY VIEWPOINT Viewpoint

ofstakeholder

group

Example important risk

Customer

Schedule delay Below expected quality

Incomplete Scope Higher Cost

User Incomplete functionality Poor Interface Low reliability

Lack of Safety features Unsecure functions

Supplier senior management

Schedule delay Over budget

Loss of profit Below expected quality

Project manager

Changing software technology Unstable development environment Lower productivity

Schedule delay Over budget Under funding Below expected quality

Project team member

Nonstandard coding Ineffective testing Poor software design

Unstable development environment Changing software technology

From Table 2, we can see that the set of important risks

according to different viewpoints are different. There are some risks that are common across several viewpoints. For example, the schedule delay risk is a common concern of customer, supplier senior management and project manager. There are also some risks that are unique to specific viewpoint. For example, nonstandard coding and ineffective testing risks are of more important to project team members, and are often ignored by other stakeholders.

Note that although we have defined 5 stakeholder categories in our study, we are not suggesting these are the only categories. Other organizations can use more categories that suit their particular need. For example, a category of maintainer may be added. Also, our proposed methodology is not dependent on this particular set of stakeholders and can work with more categories.

III. RISK PRIORITIZATION

From the identified viewpoints, we now have different contexts for prioritization of risks. Different viewpoints will place different emphasis on various risks, and thus the prioritization according to each viewpoint will likely be

194

unique. For proper risk management, we would like to derive an overall risk prioritization with due consideration of separated viewpoints.

Let R denote the set of identified risks, R = {R1, R2, …, Rn}, Rx � R and 1 � x � n. Note that the value of Rx and the number of risks n may change with time. Let Pcu, Pus, Psm, Ppm, Ppt denote the prioritization order of R according to viewpoint customer, user, senior management, project manager, and project team member respectively. Let Ri > Rj represent risk Ri has a higher priority than risk Rj. Then, the prioritization according to viewpoint y is Py = (Ry1 > Ry2 > …> Ryn), where y � {cu, us, sm, pm, pt}.

Let the set of top m risks identified from viewpoint y be RTy = (RTy1, RTy2, ..., RTym), where y � {cu, us, sm, pm, pt}, and m� n. For example, given a set of risks R=(R1, R2, R3, R4, R5), the customer may order the top 3 risks as follow: Pcu=(R2>R3>R1), while the supplier senior management may come up with the following prioritization order: Psm=(R1>R3>R4).

For any two given viewpoints y and z, with RTy and RTz respectively, there are 3 possible relationships:

Case 1: Complete overlap with RTy=RTz Viewpoint y and z identified the same set of top m risks.

Case 2: Partial overlap with RTy � RTz � � There exist some risks that are in both RTy and RTz, and also some risks in RTy that are not in RTz and vice versa.

Case 3: No overlap with RTy � RTz =� There is no overlap of the top m risks according to viewpoint y and z.

We need some method to generate the overall risk prioritization based on the separated prioritizations of different viewpoints. Next, we introduce three methods for doing this. Our methods follow the common scoring methods used in risk management [9], enhanced with a weighted component for viewpoint. As pointed out in [10], we must be careful when using these scoring methods.

For the sake of exposition, we will first use a simple example of complete overlap to illustrate our methods. Later, we will consider the other two cases and show that the proposed methods can also be applied with no change.

Example 1: Suppose three top ranked risks have been identified according to each viewpoint and their prioritizations are shown below:

Pcu = (Ra > Rb > Rc) Pus = (Rb > Rc > Ra) Psm = (Ra > Rc > Rb) Ppm = (Ra > Rc > Rb) Ppt = (Rb > Ra > Rc)

RTcu =RTus=RTsm=RTpm=RTpt=(Ra, Rb, Rc) Method 1: Combined Rank (CR) method

In this method, we first determine the prioritization of each risk according to different viewpoints. We assume that the stakeholders can prioritize risks on an ordinal scale in order of

their importance based on their respective viewpoint. A common method to do this is by paired comparison [11].

We use the Borda method [12]. For a list of m risks, the top ranked risk will be assigned a priority of m, the second ranked risk will be assigned a priority of m-1, etc. All those risks which are not assigned a priority are considered not important and assumed to have a priority value of 0. Then, we work out the combined priority score based on the separated prioritization order. The overall score OSi for risk Ri is computed by summing the separated priority scores Ki from all viewpoints.

OSi(CR) = Kicu + Ki

us + Kism + Ki

pm + Kipt (1)

where Ki

y represents the priority according to viewpoint y. Then, the risk with the highest overall score is the top priority risk, the risk with the second highest overall score is the second priority risk, and so on. Table 3 shows the result for Example 1. The final ranking is Ra > Rb > Rc.

TABLE 3. RISK RANK BY VIEWPOINT, OVERALL SCORE AND RANKING FROM CR METHOD FOR EXAMPLE 1

Risk Risk Priority by Viewpoint Overall score

OS(CR)

OverallPriorityPcu Pus Psm Ppm Ppt

Ra 3 1 3 3 1 11 3 Rb 2 3 1 1 3 10 2 Rc 1 2 2 2 2 9 1

The combined rank method is a simple method, as we

implicitly assign equal weight to all viewpoints. For example, the importance of viewpoint Pcu is the same as that of Ppt . However, to be realistic, certain viewpoint may be given higher importance. For example, the view of customer may be more important than that of the project team member. Thus, we develop a new method that considers the relative weight of viewpoints next. Method 2: Weighted Viewpoint Combined Rank (WCR) method

In the weighted viewpoint combined rank method, we first assign weight to each viewpoint, then we calculate the overall score of a risk by summing the weighted scores of all viewpoints. The set of assigned weights is named viewpoint weighting scheme (VWS).

VWS: (Wcu, Wus, Wsm, Wpm, Wpt)

where Wy represents the weight of viewpoint y. Note that the sum of all the weights is 1.

Wcu+ Wus+ Wsm+ Wpm+ Wpt= 1 (2)

There are many ways to assign the weights. A practical approach may assign the weights so that they satisfy the following relationship:

Wcu �Wsm �Wpm � Wus � Wpt (3)

195

That is, the customer and supplier senior management

viewpioints are given the highest weight, the user and project team viewpoints are given the lowest weight, and the weight for project manager should not be lower than that of user and project team member. The rationale being that the customer and senior management tend to focus on higher level concerns such as schedule delay or cost overrun. Also, being at a higher position in the organizational structure, they control the budget and other resources that the project must rely on. On the other hand, the concern of the user and project team members are more specific and at a lower level, which tend to have a relatively smaller impact on the overall project. As the project manager has the overall responsibility of the project, his viewpoint should have a higher weight than that of the project team members and also that of the user.

The overall score OSi is computed as follows:

OSi(WCR) =WcuKicu+WusKi

us+WsmKism+WpmKi

pm+WptKipt

(4) where Ki

y represents the priority according to viewpoint y. Table 4 shows the result for Example 1 when we apply the

WCR method with the viewpoint weighting scheme VWS1: VWS1: (Wcu=0.3, Wus=0.1, Wsm=0.3, Wpm=0.2, Wpt=0.1)

The final ranking is Ra > Rb = Rc.

�TABLE 4. RISK RANK BY VIEWPOINT, OVERALL SCORE AND RANKING FROM WCR METHOD WITH VWS1 FOR EXAMPLE 1

Risk Risk Priority by Viewpoint Overall

score OS(

WCR)

Overall

priority

Pcu

(Wcu=0.3)

Pus

(Wus=0.1)

Psm

(Wsm=0.3)

Ppm

(Wpm

=0.2)

Ppt

(Wpt=0.1)

Ra 3 1 3 3 1 2.6 3 Rb 2 3 1 1 3 1.7 2 Rc 1 2 2 2 2 1.7 2

Method 3: Weighted Viewpoint Weighted Score (WWS) method

In this method, we assume that the risk scores on interval scale can be provided. In other words, we assume the practitioner can assign a rating (scale from 0 to 10), where a 0 rating represents that the risk does not need to be considered further. We also assume that the practitioner will assign a relative weight to each risk within a viewpoint. Naturally, any risk assigned a relative weight of 0 is considered not important. In particular, we are using the weighted scoring method combined with weighted viewpoint to help to prioritize the risks.

The weighted risk score is computed by multiplying the risk rating and relative weight:

Si

y = Tiy Wi

y (5)

where Tiy denote the rating given to Ri under viewpoint y and

Wiy denote the relative weight assigned to Ri under viewpoint

y. The sum of all the relative weight under a viewpoint y should be 1. That is,

��

m

i 1

Wiy = 1 (6)

For example, assume the rating and relative weight for

viewpoint Pcu of Example 1 are given respectively in column 2 and 3 of Table 5, then the weighted risk score is given in column 4.

TABLE 5. WEIGHTED RISK SCORE OF VIEWPOINT PCU

Risk Rating Relative Weight

Weighted Score, Sicu =

rating x relative weight Ra 7 0.5 3.5 Rb 6 0.3 1.8 Rc 2 0.2 0.4

The WWS method also uses the viewpoint weighting

scheme. The calculation of the overall prioritization is similar to the WCR method, except we use the weighted risk score, rather than the priority.

OSi(WWS) =WcuSi

cu+WusSius+WsmSi

sm+WpmSipm+WptSi

pt (7) Wcu+ Wus+ Wsm+ Wpm+ Wpt= 1 where Si

y represents the weighted risk score according to viewpoint y, and Wy represents the weight of viewpoint y.

Table 6 shows the results of applying the WWS method with viewpoint weighting scheme VWS1 to Example 1. The final ranking according to WWS is Ra > Rb > Rc

TABLE 6. WEIGHTED RISK SCORE BY VIEWPOINT, OVERALL SCORE AND RANKING FROM WWS METHOD WITH VWS1 FOR

EXAMPLE 1 Risk Weighted Risk Score by

Viewpoint Overall

score OS(WWS)

Overall priority

Pcu Pus Psm Ppm Ppt

Ra 3.5 0.75 3.5 3.5 0.5 2.925 3 Rb 1.8 4 0.4 0.9 4 1.64 2 Rc 0.4 1.25 1.5 0.8 1 0.955 1

We next show that the three prioritization methods (CR,

WCR, and WWS) can be applied to the other two possible cases: partial and no overlap. For these two cases, some risk Ri that appears in viewpoint y may not appear in another viewpoint z. When we apply the CR method or the WCR method, this risk Ri will not have any priority under viewpoint z and will be assigned a priority value of 0. Thus, we can continue to use (1) and (4) in calculating the overall score OS.

For the WWS method, Ri will not be assigned a weight under viewpoint z and thus its weighted risk score will be 0 (Si

z = 0). Again, we can continue to use (7) to calculate OS.

196

IV. A CASE STUDY

Our case study comes from a benchmarking project for a public organization. In particular, this organization wants to know whether the cost of their computer hardware and software procurement is competitive with that offered to other commercial companies. At the end of each quarter, about 30 recently purchased items will be selected and invitation for quotation will be sent out to a number of vendors. The resulting bids will be analysed and compared to the purchased prices. This benchmarking exercise is outsourced to our research lab. We are given six weeks to complete each benchmarking exercise.

Overall, a common issue with the proposed prioritization methods is that they may produce equal ranked risks. This issue can only be solved if we select the viewpoint weighting scheme and the relative weights carefully.

We also observe that the ranking orders from different methods are slightly different. Based on the ranking from these methods, we cannot tell which method is better than the other. However, applying the CR method on the overlap and partial overlap cases, the top ranked risks will come from the top ranked risks of each viewpoint. We will not be able to come up with a unique prioritization order. Thus, we believe that the WCR method is better, as it takes into account the importance of different viewpoints.

For the WWS method, we find that the assigned relative weighting scheme is often ad-hoc and the stakeholders have difficulty in coming up with their values. It is more difficult to use, compared to the WCR method. Thus, overall, we believe that WCR is a more practical approach.

V. CONCLUSION In this paper we first propose the concept of viewpoint-

based assessment of risks based on the notion of viewpoint used in requirements engineering and then present three methods (CR, WCR, and WWS) for generating the overall risk prioritization based on the separated viewpoint-based prioritizations. These methods use the common scoring techniques (both weighted and non-weighted). Applying the proposed methods to a case study, we illustrate their relative strengths and weaknesses. We observe that the prioritization order is dependent on two key factors: the number of viewpoints which select a specific risk as the top rank and the viewpoint weighting scheme used. Also, we may obtain equal ranked risks from these methods. Based partly on our case study, it appears that the weighted viewpoint combined rank method is more practical than the other two proposed methods. It is also relatively easier to use when compared to the WWS method.

We have used a small case study to illustrate our approach. There are certainly more work needs to be done. Our follow-up study includes:

1) We will test our proposed methods on more projects. In particular, we would like to assess the practical value of

these methods when applied to some medium to large-scale software projects.

2) In the case study, we have focused on project risks and did not consider product risks. If we include other types of stakeholders, we may be able to identify some product risks and then check whether our proposed methods can also be applied to them.

3) We will try to develop other prioritization methods that can compete with the WCR method.

4) An important factor that we have not considered is the dependency that may exist among the risks [17]. We will analyse its effect on risk prioritization.

ACKNOWLEDGEMENT This research is partly supported by Hong Kong Polytechnic University grant GYK27.

REFERENCES [1] R. Williams, G. Pandelios and S. Behrens, Software Risk Evaluation

(SRE) method description (version 2.0), Software Engineering Institute, 1999

[2] Project Management Institute, A Guide to the Project Management Body of Knowledge, 2008 Edition,

[3] IEEE Std 1540-2001, IEEE Standard for Software Life Cycle Processes – Risk Management, IEEE ISBN 0-7381-2834-1, 17 March 2001

[4] B. Boehm, “Software Risk Management: Principles and Practices”, IEEE Software, 8(1), pp.32-41, January 1991

[5] P. Darke, G. Shanks, “Stakeholder viewpoints in requirements definition: a framework for understanding viewpoint development approaches”, Requirements Eng, 1(2):88–104, 1996

[6] G. Kotonya, I. Sommerville, Requirements engineering with viewpoints. Software Eng J, 11(1):5–11, 1996

[7] JCSP. Leite, P.A. Freeman, “Requirements validation through viewpoint resolution”, IEEE Trans Software Eng, 17(12):1253–1269, 1991

[8] B. Nuseibeh, J. Kramer, and A. Finkelstein, “A framework for expressing the relationships between multiple views in requirements specification”, IEEE Trans on Software Eng, 20(10):760-773, Oct. 1994.

[9] J. D. Van Vactor, “Risk Mitigation Through a Composite Risk Management Process: The U.S. Army Risk Assessment”, Org. Dev. J., Vol. 27, no. 4, pp. 84-97, 2009

[10] D. Hubbard, E. Evans, “Problems with Scoring Methods and Ordinal Scales in Risk Assessment”, IBMJ. Res. & Dev., vol. 54, No. 3, pp. 2:1-2:10, May/June 2010

[11] L.L. Thurstone, “A law of comparative judgement”, Psychological Review, 34, pp. 278–286, 1927

[12] http://en.wikipedia.org/wiki/Borda_count [13] W. G. Stillwell, D. A. Seaver, W. Edwards, “A comparison of weight

approximation techniques in multiattribute utility decision making”, Organizational Behavior and Human Decision Processes 28, 62–77, 1981

[14] F. H. Barron, B. E. Barret, “Decision quality using ranked attribute weights”, Management Science, 42, 1515–1523, 1996

[15] G.A Toth, “Automated method for identifying and prioritizing project risk factors”, Automated Software Engineering, Vol 2, No 3, pp. 231-248, 1995

[16] R. Tavakkoli-Moghaddam, S.M. Mousavi, H. Hashemi, “A fuzzy comprehensive approach for risk identification and prioritization simultaneously in EPC projects”, Risk Management in Environment, Production and Economy, 2011

[17] T.W. Kwan, H.K.N. Leung, “A Risk Management Methodology for Project Risk Dependencies”, IEEE Transaction on Software Engineering, 37(5), 635-648, Sept 2011.

197