[IEEE International Conference on Global Software Engineering (ICGSE 2007) - Munich, Germany (2007.08.27-2007.08.30)] International Conference on Global Software Engineering (ICGSE

Virtual Quality Assurance Facilitation Model

Rajesh Agarwal; Pinakpani Nayak; Manickam Malarvizhi; P Suresh; Nina Modi Tata Consultancy Services Ltd.

[email protected]; [email protected]; [email protected]

Abstract

The IT industry’s annual growth rate of ~40%

has put immense pressure on the Software Quality Assurance function. Motivated by this challenge, we evaluated various options, before adopting virtual quality assurance facilitation at TCS. In the As-Is model, TCS has a traditional quality organization wherein a small number of full-time software quality analysts facilitate several projects each, with reporting independent of the delivery organization. In the To-Be model, we envisaged many part-time SQAs facilitating a smaller number of projects. Line reporting of the SQAs was to be left within the delivery organization. Evaluation of performance of this model has been very encouraging. Internalization of processes, building competency, and maintaining process stability with reduced cost were some key benefits. This paper describes the genesis, results, benefits, and challenges as experienced during the pilot and suggests the way ahead for deploying the virtual SQA function in large, mature IT organizations. 1. Introduction

Software Quality Assurance (SQA) is defined as a planned and systematic approach to the evaluation of the quality of and adherence to software product standards, processes and procedures. SQA includes the process of assuring that standards and procedures are established and followed throughout the software life cycle. Compliance with agreed-upon standards and procedures is evaluated through process monitoring, product evaluation and audits. The software development and control process includes quality assurance approval points, where an

SQA evaluation of the product may be done in relation to applicable standards.

Tata Consultancy Services (TCS) is a CMMi and PCMM Level 5 company. It has an elaborate quality organization to define, deploy and verify processes. For the deployment of processes and to perform the SQA function, a Quality Assurance Group (QAG) is established. Quality Assurance Facilitators (QAFs) facilitate the deployment of processes within projects. 2. The traditional SQA model

In a traditional quality organization, the quality head reports directly to the management representative. In the entire chain of the quality function, reporting of associates in the quality group is typically isolated from the delivery organization. The intent is to keep the quality group away from the pressures of the delivery function and to maintain independence in the case of a difference of opinion between the functions. Figure 1 presents the organization chart of the quality group at TCS, and Figure 2, a typical organization structure of the QAG within a TCS branch.

TCS’s existing SQA model at the branch level comprises a core group of QAFs led by the branch Quality Coordinator (QC), reporting to the Quality Group Leader (QGL), who in turn, reports to the Corporate Quality Head (Figure 2).

A QAF (or more than one, depending on the size of the business group) is assigned to a particular business group. Each full-time QAF performs the SQA function and, on an average, facilitates about 12 to 18 projects. Senior QAFs are given the additional responsibility of being the Single Point Of Contact (SPOC) for a business group and interacting with the delivery head.

International Conference on Global Software Engineering(ICGSE 2007)0-7695-2920-8/07 $25.00 © 2007

A pool of reviewers is identified from the delivery teams. Technical reviewers carry out independent reviews on artifacts created by projects.

Internally certified auditors conduct audits on projects to verify process compliance. Risk review is performed by a pool of management representatives.

Figure 1. Quality Organization in TCS Figure 2. Quality Assurance Organization in a TCS Branch

Quality Group LeaderQuality Group Leader

CEOCEO

Exec Vice President(Quality)


Branch SEPGBranch SEPG Branch QAGBranch QAG Branch BAGBranch BAG

Definition Deployment Verification

Corporate Quality HeadCorporate Quality Head

BAG: Branch Audit Group QAG: Quality Assurance GroupSEPG: Software Engineering Process Group

Quality Group LeaderQuality Group Leader

CEOCEO



Branch SEPGBranch SEPG Branch QAGBranch QAG Branch BAGBranch BAG

Definition Deployment Verification

Corporate Quality HeadCorporate Quality Head

BAG: Branch Audit Group QAG: Quality Assurance GroupSEPG: Software Engineering Process Group

EVP

Delivery Head

Group Leader

Account Mgr

Project Leader

Project Team

Quality Group Leader

Quality Coordinator

SPOC - QAFs

QAF QAF QAF

Pool of Reviewers

CQH RM

CEO

Quality Organization Delivery Organization

CQH: Corporate Quality Head QAF: Quality Assurance FacilitatorRM: Regional ManagerSPOC: Single Point Of Contact

Functional Line

Reporting

EVP

Delivery Head

Group Leader

Account Mgr

Project Leader

Project Team

Quality Group Leader

Quality Coordinator

SPOC - QAFs

QAF QAF QAF

Pool of Reviewers

CQH RM

CEO

Quality Organization Delivery Organization

CQH: Corporate Quality Head QAF: Quality Assurance FacilitatorRM: Regional ManagerSPOC: Single Point Of Contact

Functional Line

Reporting


3. Need for the new SQA model 3.1. Growth of the organization

The IT industry is growing at a phenomenal pace and TCS, one of the leaders in this sector, is growing at the rate of 40% year-on-year. This growth puts immense pressure on the support functions including SQA.

Apart from the cost of operation, the number of QAFs required to perform the SQA function needs to increase in proportion to the growth in the number of projects and their sizes; staffing the QAG on an ongoing basis thus becomes a challenge. (Consider an organization where roughly 600 projects get executed — about 40 full-time QAFs would be required). 3.2. Rigor of the SQA model

TCS’s iQMS (Integrated Quality Management System) envisages well-integrated project management, execution and quality processes. In line with this, cross-pollination between delivery and quality is sought by rotating associates between the quality group and delivery.

The iQMS is model independent. It integrates the key practices required by any model and thereby ensures that the SQA function focuses on processes rather than the models to which they adhere to.

The rigor of TCS’s iQMS requires capable people, at the same time the growth of the industry and paucity of resources has made it challenging to staff QAG with people who can mentor and facilitate projects. 3.3. Internalizing of quality

With a dedicated SQA function, the experience was that at times project teams believed that quality processes were different for project management and execution processes, creating a dichotomy. This created challenges with respect to internalizing quality processes as part of the overall execution process among the delivery teams.

4. The new virtual SQA model

To address these challenges, a new model with virtual team members was thought of. The objective was to internalize quality processes, build a team of quality facilitators which would be self sustaining and value adding to project teams, ensure process compliance with rigor and have speed in execution.

The proposed new model modifies the core QAF structure by entrusting the quality facilitation activity within the business groups. According to this model, each business group (Figure 3) has a Quality Manager (QM) supported by a group of Virtual Quality Assurance Facilitators (VQAF). It needs to be noted that while the role of QM did exist in several mature relationships, it was not prevalent across the board and was typically customer-driven.

This concept of a virtual team performing the SQA function is also not new, as TCS always had a virtual pool of reviewers. This model extends the concept to deploy virtual SQAs within the delivery organization. 4.1. Virtual quality assurance facilitators

The VQAFs are Project Leaders (PLs), backup PLs or senior associates who are experienced as Project Leaders. The responsibilities of virtual team members of the QAG will be to perform the SQA function and provide iQMS facilitation to projects. These tasks may include facilitation of processes, conducting process reviews, and reporting project health (process and product). Data from TCS’s existing SQA model evinced that, on an average, one VQAF can facilitate about three projects, putting in approximately four hours of effort per week. To avoid any conflict of interest VQAFs are deployed across groups.

It may be noted that this is an additional role of the virtual team member, and the contribution to the project as per primary allocation will have to continue without any deterioration of content, quantity and quality. Why an individual would take up these additional responsibilities will be clarified in the section on benefits.

The formal reporting of this individual will continue to be as per the delivery organization structure. However functionally, the VQAF will have a dotted line reporting to the QM.


Figure 3. QM/VQAF Model in a Delivery Organization 4.2. Quality manager

The QM will report to the business Group Leader (GL) or the Delivery Head (DH) depending on the size of the group or delivery center. The role of the QM can be dedicated full-time or part-time, depending on the size of the business unit.

QMs are responsible for coordination and act as the hub for the business group for all VQAFs, monitor the process health of the entire Business Group, conduct Project start up meetings, complete project wind ups, update Business Leaders on project health, carry out risk analysis, provide lead indicators for red projects, take preventive action to mitigate recurring problems, provide data to QAG and Business as required and act as a bridge between QAG and the Business Group. 5. Deployment of the new SQA model

The model was piloted in some mature business groups at one of the large branches of TCS. The selection criteria for the pilot were the maturity of the business units and the stability of execution processes in that business. When the SQA function is being facilitated from within delivery, customer-specific processes, domain practices and norms are

easily facilitated by the VQAFs rather than putting a learning overload on the core QAFs.

Each GL and the respective Account Managers (AM) identified the QMs/VQAFS after considering the selection criteria and the abilities of the associates. There was a detailed three week transition and training plan to hand over the SQA activities to the VQAFs and QMs from the existing QAF team. FMEA (Failure Modes and Effects Analysis) was prepared to understand the risks and the mitigation plans. The pilot started from April 2006 in a phased manner across business groups. 6. Effectiveness of the new SQA model

After four months of deployment within a delivery center, a survey was planned to study the effectiveness of the new model and substantiate the same with other results of iQMS analysis.

6.1. Survey analysis

The survey respondents were grouped into three categories, namely: the delivery management category consisting of DH, GLs and AMs, the Quality Managers, and the VQAFs.

The survey results were encouraging, with the main observation being that the new model is value adding

QM

Functional LineA B

AM1

PL

VQAF 1 -N

PL

VQAF 1 -N

AM2

PL

VQAF 1 -N

PL

VQAF 1 -N

A B

Group Leader

Delivery Head

QC

Reporting

AM: Account ManagerPL: Project LeaderQC: Quality CoordinatorQM: Quality ManagerVQAF: Virtual QAF

QM

Functional LineA B

AM1

PL

VQAF 1 -N

PL

VQAF 1 -N

AM2

PL

VQAF 1 -N

PL

VQAF 1 -N

A B

Group Leader

Delivery Head

QC

Reporting

AM: Account ManagerPL: Project LeaderQC: Quality CoordinatorQM: Quality ManagerVQAF: Virtual QAF


and effective, the delivery management does not want to go back to the earlier model, and QMs and VQAFs are satisfied with their new role. They also found the new role value adding to them in their professional career. The model gave more control to the leadership team to control the quality health of their group.

6.1.1. Response from delivery management. When the delivery centers were approached to pilot the new model, there was resistance against the change with the main reason for resistance being that the SQA role would lose independence and there would be conflict of interest if the VQAF’s reporting was within the delivery center. Here it is interesting to note that after adoption of the new model, all respondents answered that there was no conflict of interest in project facilitation (Figure 4).

6.1.2. Response from quality managers. A few of the respondents in this category were part-time QMs who took this role as additional

responsibility (Figure 5). It was satisfying to note that QMs stated that the role was adding value to them as well to the organization.

Two factors related to digitization, coordination and follow-up were identified as pain areas and these inputs went into the new set of requirements for improvements to the digitized tools.

6.1.3. Response from virtual quality assurance facilitators. All respondents in this category are virtual members. This implies that their QAF responsibilities are in addition to their project tasks. It is therefore this category which has the most impact on the success of the model.

While VQAFs appreciate additional tasks that are value adding (e.g. knowledge enrichment, project management experience and so on) to them, an important finding from this category of respondents was their difficulty in adhering to normal service level agreements (SLAs) as applicable to full-time QAFs within the QAG (Figure 6).

Figure 4. Response from Delivery Management

Management Survey - Positive and Negative

0

10

20

30

40

50

60

70

80

90

100

Low erperceived

satisfaction of QMs

Low erperceived

satisfaction of VQAFs

Less voluntarycontribution

New Qualitymodel is

w orking w ell

Value addition of

new model

Acceptance of

new model

No conf lict of interest

%ag

e of

Res

pons

es

PositiveNegative


Figure 5. Response from Quality Managers Figure 6. Response from Virtual Quality Assurance Facilitators 6.2. iQMS analysis

At TCS, project health is tracked through various dashboards, using Red, Amber and Green (RAG) criteria.

Red Projects lacking rigor with respect to processes and projects in crisis.

Amber Projects with process compliance issues which may lead to chronic issues and high risks.

Green Projects with no issues.

The effectiveness of the new model was tested by analyzing the age and percentage of Red Amber Green status projects within the Business Units which implemented the new model.

Project health status as shown in Figure 7 does not indicate any increase in red projects. Variation in Amber projects also does not show any apparent change.

A comparison was also sought on the distribution of issues on the basis of CMMi key process areas as

QM Survey - Positive and Negative

0

10

20

30

40

50

60

70

80

90

100

Lack of Digitization

Time spent on

Follow up

Lack of Rew ard and Recognition

Know ledgeEnrichment

QM role isvalue adding

Ease of coordination w ith VQAF

Willrecommend Best Friend

No conflict of interest

Exposure to senior

management

Good Model

%ag

e of

Res

pons

esPositiveNegative

VQAF Survey - Positive and Negative

0

10

20

30

40

50

60

70

80

90

100

Lack of Digitization

Time spent on

Follow up

SLA compliance

Lack of Rew ard and Recognition

VQAF Role is value adding

Time Spent w ithin

Planned Limit

No conflict of interest

Ease of coordination

w ith QM

Will recommend Best Friend

Know ledge Enrichment

Good Model

%ag

e of

Res

pons

es

PositiveNegative


well as the count of issues and audit NCs opened per project before and after the implementation of the model. There was no impact on these numbers.

Trends of some of the key delivery performance metrics of the branch (Figure 8) also indicate that the model has no impact on delivery capability. The reduction in bad fixes is actually a positive sign, but it cannot be ascribed to the change in the SQA model alone.

The trends imply that the VQAFs and QM are efficiently and proactively facilitating projects to address quality related issues and there is no dilution of the SQA function in the organization.

The performance of the model continues to be monitored as part of iQMS analysis for any correction and tuning if required.

Figure 7. Trend Analysis – Project Health Dashboard Figure 8. Key Delivery Metrics (a large relationship)

Bad Fix

0

0.2

0.4

0.6

0.8

1

Q1 Q2 Q3

Old New

On Time Delivery

5060708090

100110120

Q1 Q2 Q3

Old New

Defect Free Delivery

5060708090

100110120

Q1 Q2 Q3

Old New

SLA Compliance

80

90

100

110

120

Q1 Q2 Q3

Old New


7. Benefits of the new model

Having established that the new model did not result in a loss of effectiveness of the SQA function, three important benefits are discussed here. Other benefits which may be realized depending on the size, maturity and management will of the relationship and digitization of processes are not covered here. 7.1. Managing growth

The growth of the industry (Section 3.1) was one of the prime motivators for adopting this model. In the old model (Table 1), each QAF facilitated 17 projects. With the deployment of the QM/VQAF model, the growth of the branch was not only adequately managed but the number of projects per

full time equivalent (FTE) was reduced. The table also shows that the number of projects per FTE has reduced, allowing more facilitation time per project. 7.2. Internalizing of quality

On an average, the term of a core QAF in QAG is about a year as against about 3-4 months for a VQAF. Therefore, while in the earlier model, staffing of the QAG was a constant challenge, VQAF positions are filled relatively quickly.

An advantage of having the SQA function within the delivery organization is that industry practices and norms are easily facilitated by the VQAFs. The earlier model required core QAFs to be domain and industry experts for all the projects they facilitated.

Table 1. Increase in FTEs in proportion to growth

Role No. of associates FTE No. of projects No. of projects/FTEQM 0 0 0 0VQAF 0 0 0 0QAF 25 25 425 17Total 25 25 425 17

Role No. of associates FTE No. of projects No. of projects/FTEQM 21 15.1VQAF 130 18.7QAF 10 10 139 13.9Total 161 43.8 577 13.2

438 13.0

New Model (Current)

Old Model (March 2006)

7.3. Building organizational competency

Careful analysis of the data in Table 1 also indicates another resultant benefit of the new model. In the traditional model, about 34 quality group members would be required to facilitate 577 branch projects. This meant that at any point in time only 34 employees would gain experience in the SQA function as against 160 employees benefiting from this exposure. This is big win for any organization.

7.4. Removing the perception of loss of independence

A concern of delivery managers prior to the piloting of the new model was related to independence and therefore the possible dilution of the SQA function. Analysis of performance before and after does not show any such behavior. Further the responses from the delivery managers during the survey suggest that these fears have been addressed. This is primarily due to the fact that project reviews,


external quality assurance reviews and audits continue to be carried out by individuals who are not responsible for delivery of the reviewed project. 8. Going forward

No model is perfect or suits all situations. This model too has limitations, which were identified during the months of its deployment.

Some of the critical to success factors are: 1. Stable and mature delivery processes 2. Digitization of SQA tasks 3. Rewards and recognition to attract the best

talent as VQAFs 4. Rotation policy for VQAFs 5. Revised SLAs This model was piloted with success in some

mature business groups and provides an option for deployment of SQA function in an organization.

At the cost of forecasting, the authors opine that as organizations mature, external process

verification will soon become a thing of the past. The process of self declaration will come into vogue with SWAT teams to support projects which declare risks in their functioning. 9. Conclusion

This new model is working as per plan and improving continuously as it matures. Contrary to general belief, the integrity of process compliance is not compromised. Acceptance by the leadership team is also growing as they see the benefits. Intangible savings from this model as a result of internalization of QA processes and creation of organizational competency surpasses the financial savings. Overall, this model is the way ahead for deploying the SQA function in large, mature IT organizations. Acknowledgements The quality group of the branch where this new model was deployed acknowledges all delivery managers for their continuous support and commitment to the rationale behind this model.


Documents

[IEEE International Conference on Global Software Engineering (ICGSE 2007) - Munich, Germany (2007.08.27-2007.08.30)] International Conference on Global Software Engineering (ICGSE