Abstract: Many organizations have turned to outsourcing to cut

costs. But how can you gain a strategic advantage from an

outsourcing initiative, not just cost savings? This white paper will

provide executives and managers of software product and service

companies with a ―How To‖ guide and best practices for integrating

independent Quality Assurance and testing across the product

lifecycle to shorten software development cycles.

Technology companies are challenged to innovate faster than their competition, to deliver their applications to

market first, and to re-balance their development priorities in order to accelerate revenue generation.

Business leaders often view software quality as a luxury - something that can be sacrificed, if necessary, for more

functionality, faster development, or lower costs. However, in practice, software development organizations that have

a firm commitment to quality can actually improve development cycles, reduce costs, and add new features with

greater ease. Organizations that develop low-quality software, whether for sale or for internal use, are always looking

backward, spending time and money on fixing defects in "finished"

products. In contrast, an organization that builds-in product quality

from the beginning can be forward-looking and innovative; it can

spend its resources on pursuing new opportunities.

According to U.S. Department of Commerce, software bugs cost

the U.S. economy an estimated $59.5 billion per year. $22.2 billion

could be eliminated with improved testing and earlier identification

of errors.

Customer expectations of flawless execution have become par for

the course. In today’s competitive economy, no software vendor

can afford the tarnished reputation or unhappy customers that

result from preventable software | defects. But when it comes down to traditional methods for assuring software

quality, companies are finding it increasingly difficult to give up the valuable time and strategic focus that get

consumed by resource-intensive internal test efforts—a problem that grows larger as development projects advance

in both scope and complexity. It’s no surprise that many companies are turning, with increasing popularity, to

outsourced methods of software quality assurance as a way to ensure the integrity of their products, the satisfaction

of their customers, and the long-term success of their growing development initiatives—all while keeping costs down.

As a software company, you face a unique set of development challenges— more platforms and configurations to

support, more releases to sustain, more testing to be completed, and more products to integrate.

Historically, QA is managed at a discrete project level and not coordinated across the product portfolio. As a result,

there are several challenges:

Limited strategic direction for QA at an organizational level

No standardization of tools/processes/test environment

No centralized reporting or service execution

A limited view of function level progress/quality

Product-release delays due to excessive testing time

A study commissioned by the U.S.

Department of Commerce's National

Institute of Standards and Technology

(NIST) found that software defects

cost the U.S. economy almost $60

billion annually. $22.2 billion could be

eliminated with improved testing and

earlier identification of errors.

Lack of quality testing resources, preventing quick application delivery

Product-release delays due to inability to quickly ramp-up QA team when needed

Absence of real-time data for managers to evaluate progress of testing

You need tools and practices that accelerate the development process to maintain and extend existing code as well

as to develop and deploy new code—while tightly managing the costs associated with numerous members of your

development team spread out across multiple locations. Your bottom line is dependent on your development team’s

ability to produce quality software.

When it comes to product lifecycle management, there are several primary objectives:

Shorten time-to-market timeframes

Software companies should rapidly narrow the gap between what the customer needs and what the

software delivers by increasing functionality, adaptability, usability, and manageability.

Deliver increased product feature sets

There should be a strategic release management plan that optimizes usage of the product

engineering and QA team.

Improve product quality

Software companies may use the testing and

validation services of an independent team with

specialized skills that stays focused on delivering

a zero-defect product.

Constantly increase software development

efficiencies

To succeed in the next economic cycle in the software business, companies have to increase

efficiencies and productivity, and create and consolidate software assets into reusable components

and blocks.

Increase market share and customer/partner retention byfocusing on the strategic activities

and the core business

Software companies can leverage a thoughtful and balanced offshore strategy across the entire

product lifecycle to keep key people focused on strategic activities such as product management,

marketing, sales, and customer service.

The emerging technology leaders are

constantly improving speed, quality,

and cost parameters at each step of

the product lifecycle.

The Informatics Computer School's Software Engineering course-text defines Software Quality as:

Conformance to clearly stated functional and performance requirements

Conformance to clearly documented development standards

Conformance to implicit characteristics that are expected of all professionally developed software

For any organization, defining what you mean by "quality" is an important first step. Too often, software development

organizations operate with a loose, general notion of quality and

tolerate defects that most engineering disciplines would not allow.

In contrast, a solid definition of quality that all team members

understand and accept promotes thoroughness and attention to

detail. In the domain of business applications, we can best define

quality in terms of the target audience: the software users.

Development organizations with a quality focus know that a

"quality" application must do more than simply provide correct

results without crashing. Does the application meet stakeholder

requirements? Is it usable? Secure? Scalable? Reliable? Easily

maintained? Easily extended? Simple to monitor?

James Juran's definition provides a good starting point: quality is

"fitness for use." Juran goes on to say that a product is not high-quality unless it adds value for both the consumer

and the manufacturer.

Put slightly differently, quality encompasses added value plus attention to detail. Both a luxury car and an entry-level

car will get you from point A to point B. But the luxury car offers features and capabilities that go beyond the

essentials of transportation: usability, safety, comfort, reliability, and so on. Product quality also reflects the process

behind the product. In the software world, a high-quality process can keep development organizations from losing

time reworking, re-factoring, and rewriting software. These organizations can produce more innovative and creative

products because they have more time to think about adding value and quality details.

Achieving quality results requires applying a high-quality process throughout development, integration, and testing.

This applies equally to projects involving packaged or homegrown applications, upgrades or greenfield development

work, and extending, integrating, and modernizing legacy applications (see Figure 1).

On average, developers make 100 to

150 errors for every thousand lines of

code they write. Even if only a small

fraction—for example 10 %—of these

errors are serious, then a relatively

small application of 20 thousand lines

of code will have roughly 200 serious

coding errors.

Figure 1: All modes of software development require the same attention to quality.

But the bottom line of software quality is conformance to the user's needs⎯or requirements,, or wants. The point is,

any definition of software quality must take into consideration the end-user's perspective. An application may do all

that it was designed to do, and do it according to professional standards, but if users hate using it, find it difficult,

counterintuitive, or error-prone, then its quality is at best questionable.

Being free of defects is not enough. For a product to achieve quality, it must also provide exceptional functionality,

usability, and compatibility.

Software Quality Assurance cannot be treated by executives, managers, or engineers as one additional "Task" on

the way to the final launch, or a thing that can be taken care of

as the project proceeds, along the development and release

process. Team members closely associated with the application

are less likely to find the problems that an external party would

find easy to spot and report. Independent testing efforts ensure

that your product is tested by individuals who will be just as

objective and unforgiving as your customers in the case of a

potential defect.

This is simply a fact, and not a judgment of or reflection on the

developers' talents. An individual who spends the most time

working on a project intended for an audience cannot simply step back and look at it as though through that

audience's eyes. The creator is too familiar with his work to view it objectively.

New

Development

Legacy

Systems

Packaged

Applications

Design

Develop

Test

Deploy

Extend

Expose

Upgrade

Customize

Expose

Test

Deploy

Integrated

"Rigorous inspections can remove up

to 90% of errors before the first test

case is run, but rigorous inspections

should not replace testing" - Robert L.

Glass, "Facts and Fallacies of

Software Engineering"

But this is why Quality Assurance experts exist⎯just asprofessional proofreaders, copy editors, test audiences, and

software do. These independent parties bring much-needed objectivity to the appraisal of a software development

project.

It is crucial that software Quality Assurance be independent of project management for it to be completely effective.

Outsourcing software Quality Assurance to an independent, unbiased source is the most effective way to achieve the

objectivity that a project needs.

When it comes to the practice of independent verification and validation, an independent QA team should be the one

final check-off point before allowing a system to go into operation; how it is done systematically and organizationally

varies, however.

The standardization of software engineering processes, margin compression, and competition are compelling

technology-centered companies to face the task of delivering more complex application development with limited and

decreasing time frames. Software leaders are looking at outsourcing processes across the entire software

development lifecycle, from engineering, to quality assurance and testing, to maintenance and porting.

Software Quality Assurance outsourcing can take a number of paths, testing for different capabilities and performing

different analyses, depending on the demands of the project, the users, and the software itself. But an outsourced

Quality Assurance and Testing initiative carried out by seasoned QA professionals will confer certain benefits:

1. Reduced Development Cycles

2. Improved Customer Satisfaction

3. Reduced Cost of Development

4. Reduced Cost of Maintenance

Coupled with an outsourced business model and its benefits, outsourcing software QA and testing became a

strategic R&D initiative for many technology companies. The expected benefits include:

5. Improved focus of in-house resources on strategic activities and innovation

6. Reduced Total Cost of Ownership (TCO)

7. Reduced technology risks

So what is the best way to leverage QA and testing to shorten

development cycles? Build your own QA and testing department?

Create a hybrid of in-house and outsourced resources? Or

outsource entirely?

As a software vendor, innovation is the foundation of your

success. The earlier in your software development process that

you apply world-class testing and quality assurance practices, the

more efficient your process will be. Advancing other critical path activities, such as internationalization and

According to Gartner (2005), “20% of

outsourcing deals do not produce cost

savings, with 50% of all outsourcing

projects falling short of delivering the

expected value and being deemed

unsuccessful".

localization, further upstream in the product lifecycle will also serve to improve efficiency. Software makers who

manage their product lifecycles as carefully as the process-mature manufacturers in, say, the automotive industry,

are sure to be winners as the high-tech industry moves into the next economic cycle.

Outsourcing the testing activity, properly applied, can address many of these issues while relieving management of

many of the burdens of process development, staffing, and ongoing control.

The reasons for outsourcing are many and varied among potential customers. Depending on the nature and maturity

of the company, these could range from the need for limited special purpose testing to taking over quality

management of a product/application portfolio. The former might be done to address specific gaps in the testing

framework, such as a lack of environments for conducting load testing, or areas of particular risk such as security.

The latter may be advisable if your internal resources are unable or have limited capabilities to justify, explain, or

quantify existing internal testing activities. As with any business decision, the choice to outsource and/or offshore

software testing must be viewed in terms of risk and reward. While this will vary from company to company, a few

basic principles should be applied when considering such a solution.

The first step should be to articulate the goals of the initiative. The basic rationale for undertaking this should be

stated explicitly and frame the entire process. A formal strategy statement may also be valuable here to lay out a

roadmap for proceeding, articulating critical risks and assumptions, estimating resource requirements, and

determining measures of success.

Some of the possible reasons for deciding to outsource include:

1. Deficiencies in the specialized skills and

processes required to support testing

2. Problems related to poor test practices such as

excessive expenditures on maintenance and

critical failures in production

3. Lack of test infrastructure including test environments and test beds

4. Insufficient testing activity to warrant a full-time testing staff because the portfolio is too small or

there are only a limited number of releases during a year

5. Lack of facilities or expertise to conduct special purpose testing

6. Independent validation and verification of development work performed by other vendors

One other area to note where an organization might gain significant long-term benefits from outsourcing is in

implementing test automation. Manual testing and tracking is a complicated enough task and, although using

automated test tools can help improve efficiency and control, they are not an immediate panacea. Such tools must

be carefully selected to match the technology and environments used by a system. The tools must be set up, scripts

should be written and maintained by experienced professionals who know how to configure the package, set up

databases, design test artifacts, determine and enforce privileges, establish library control procedures, train staff,

and enforce standards. By engaging a service provider who knows how to set up and manage these tools, a

"...the average enterprise will

ultimately outsource 60% of

application work offshore (circa

2008/09)."— META Group

company can gain valuable experience in using them without some of the pain associated with starting up such an

initiative internally.

The final aspect of making a decision about whether or not to outsource Quality Assurance and testing is to

determine the organization’s readiness to support testing. An evaluation of existing processes, skill sets, and

infrastructure is recommended in order to set the stage for planning a test outsourcing initiative. An important part of

any such assessment is how the existing QA or testing organization will interact with the vendor.

The chapter entitled, ―Five Phases of Integrating QA Into the Entire Global Software-development Cycle" provides

step-by-step guidelines for how to design and manage your Quality Assurance and testing program.

Based on the goals articulated by management, a formal framework for moving forward should be developed. Some

potential models for outsourcing that might be considered include:

1. Total off-site outsourcing of all testing functions for the entire portfolio including offloading the

existing testing staff to the vendor.

2. Conducting outsourced testing on a pilot basis with the intention of developing internal processes or

introducing automation.

3. Partial outsourcing of special purpose testing (e.g. performance or security) or testing level (e.g.

outsourcing system integration testing but not unit testing or user acceptance testing).

4. A vendor-managed internal testing program in which all work takes place on site, applications

remain in the customer environment, test beds are kept in repositories controlled by the client, but

the test management and staffing is provided by the vendor.

5. Third party validation of outsourced products, involving independent testing of applications built by

another vendor.

Total off-site outsourcing has the advantage of reducing internal staff expenditures while improving the efficiency and

effectiveness of the testing function. It leverages the knowledge of the vendor to the largest extent. This model tends

to work best for customers who have well-documented requirements and well-educated stakeholders. In the absence

of those conditions, there may be a ramp-up period in which the vendor will have to spend time organizing

requirements and familiarizing internal staff with their roles. In addition, the relationship between the vendor and

developers must be managed carefully; delays in the delivery of software will affect testing schedules and the

amount of testing, which can be performed.

The final consideration in a pure outsourcing arrangement is the fact that the product will be under the control of the

vendor during testing. Arrangements must be made to assure the security and integrity of programs and data during

this time. A risk worth mentioning with a pure outsourcing approach is that the client can become out of touch with

the quality management of its own products. It is incumbent on the customer to review, interpret, and act on test

artifacts and test results even if the entire testing operation is run by an off-site test team.

The major benefits of running an outsourcing pilot are that targets and costs can be controlled selectively and future

strategies can be modified based on the outcomes and findings from the pilot. When choosing pilot applications, it is

usually wise to find those of enough significance and size to make the activity meaningful but not those which are

business-critical. This allows clients to learn important lessons about testing applications in their domains without

putting core systems at risk. Establishing predetermined criteria for the success of the pilot is vital to extracting the

proper lessons from it.

Partial outsourcing of special purpose tests or by testing level can be an extremely effective way for an organization

with existing test infrastructure to enhance its quality management

program.

A vendor-managed internal testing program is a hybrid business

model, which can serve multiple needs. In this scenario, all work

takes place on-site and applications remain in the customer

environment. There is greater control and much less risk of

compromising systems or data. The vendor may design and

create testing procedures or use existing processes, but they can

bring additional value to the organization by working with internal

best practices and enhancing them. The vendor may also develop

processes for controlling test beds, however it is recommended

that the repositories are designed and managed by the client’s

infrastructure team. Since the test management and staffing is

provided by the vendor, the customer does not have to invest in finding the skills needed to establish the test

infrastructure and staff the test teams. One other approach that is gaining popularity is third-party validation of

applications built by another vendor. This trend is pronounced with mid-market clients who have decided to

outsource most of their IT function. It can be a cost-effective option for organizations with little experience with

software development and testing or for those who have very few or small software projects. While it does not

contribute to the client developing its own testing processes, it is a good model for those who do not wish to invest in

that infrastructure. In addition, the test cases and test data are the property of the customer and can be reused in the

future should the outsourcing relationship be discontinued.

One issue that may arise when adopting this approach is contention among vendors. Performance of the testers can

be affected negatively by developers who do not provide good test requirements, cannot provide feedback on tests

in development, or fail to deliver on time. The client in this situation must maintain control of the process and must

take steps to assure that the vendors are working in a manner that supports the company’s goals.

At the end of the day, how does an executive determine the value of outsourced testing and whether it is worthwhile

to undertake? Potential buyers should realize that the benefits to be derived from test outsourcing may go beyond

those gained through outsourcing other functions.

Those who have survived in the tough conditions of the recent IT economy are well aware of quality issues and the

need to deliver fully tested products. The financial motivation to find and fix bugs before product release is not a new

notion, however. A study by noted industry analyst Capers Jones back in 1996 clearly showed the staggering costs

By engaging vendors with extensive

experience in a particular specialty,

such as performance testing, a

company can leverage its knowledge

and environments, reduce investment

in hard to find skills, minimize the

need for expanding test infrastructure,

and receive independent verification of

results.

associated with post-release bug detection. Post-release bug detection usually occurs when software companies

decide to cut costs by reducing testing schedules in early phases of development.

Figure 2. Cost of repairing defects across application lifecycle

The fact that closer attention to testing is vital to keeping overall long-term costs down is often obscured by near-

term budget constraints. Here we run into a paradox: more testing resources and enhanced QA requirements are

paramount to a company's success, but their budgets simply cannot support increased spending.

There is a solution to this seeming contradiction. As Steven Brier, senior analyst and managing director of research

at Roberson Stephens, notes, the secret to success in the IT landscape these days is to focus on running the core

business while effectively using outsourcing options. "It's often cheaper and faster to give it to someone else to do,"

says Brier {UPSIDE, June 2001).

In terms of quality assurance, this means that developers can stay within their core competencies by unit testing with

highly specialized debugging tools, such as jtest® from ParaSoft®, while outsourcing functionality, compatibility, and

integration testing. This allows software engineering organizations to capitalize on what they do best, i.e. write code,

while delegating the task of testing how their product functions in the complete solution stack to an outside team with

proven expertise in quality assurance.

Additionally, in many companies, because of repetitious test processes, the turnover rate for test engineers is very

high. This means that an in-house test lab must continuously spend time and money to seek and recruit skilled

workers and train them for its purposes. In comparison, outsource test organizations offer clearly defined career

paths to their employees, reducing turnover. Plus, due to the nature of the business, projects in outsourced testing

labs are always in flux, which keeps career test engineers interested in working with and discovering upcoming

technologies. As a result, these engineers are highly trained, enthusiastic, possess a wide variety of core

competencies, and are efficient in carrying out complex test projects.

It is important to understand that the savings realized through more effective testing don’t automatically fall to the

bottom line. Even if an outsourcing arrangement does directly reduce the cost of testing, most IT organizations will

probably still spend their up to their budget limits. The savings can be and usually are spent on development and

other productive tasks in support of core business functions. But some of the payback of instituting an effective

testing program is derived in other ways, including:

More efficient staff and resource management

Ability to investigate test automation in a low-risk setting

Reduction of time spent on rework and repair

Minimizing and managing product risk

By enabling the software engineering department to make more effective use of the internal resources, both testing

and development benefit. Developers can focus on the job of developing the software without worrying about

planning, designing, and executing tests. This can shorten lead times to get products to market and result in better

systems because developers will be focused on just one task - engineering - which has to be the only core

competency. Dedicated, trained testing resources who understand the overall framework of quality management will

be much more effective in constructing and implementing an organized approach to validating software. The testing

process will be more efficient and effective. First of all, highly trained specialists will be better at this task.

Effectiveness will also be enhanced since an outsourcer is more likely to provide an unbiased assessment of product

quality simply because they are operating independently from the developers. Finally, the costly and time-consuming

task of setting up and managing test environments will no longer be a factor in determining resource allocations. A

good provider of testing services should either have the appropriate hardware and software infrastructure in place or

be able to generate it at less cost and in less time.

The management of special-purpose testing will also be enhanced. Performance and stress testing can be done with

more breadth and depth. Regression testing can also be enhanced. The key to effective regression testing is to

maintain existing test scripts so that they can be re-run when changes are made. However, most companies don’t

have experience in determining which scripts to keep, how to store them in an organized fashion or when to re-run

them. A qualified service provider will have these skills on staff or make them available, when needed.

Outsourcing to a service provider who has extensive experience with automated test tools also provides a buyer the

benefits of working with a partner knowledgeable in this field. They can assist with defining the tool set to be used,

installing and configuring the product, managing it on an ongoing basis, and training staff in its use. Automated tools

can also be used to improve regression testing, as they provide facilities for storing and managing test scripts and

test data. When one considers that a significant proportion of test tools purchased are never used effectively by the

buyers due to lack of experience, the return on investment is often gained simply by using the very products they

purchase. Then there are the benefits of better product quality. Put simply, it is advantageous and economical in the

long run to test software systems effectively to assure that they meet their requirements rather than correcting them

later.

The amount of time and money spent by software engineering departments on application ―maintenance‖ can

exceed 50% of the annual budget. This calculation also does not take into account the overall cost of poor quality

which includes the time and resources required to identify defects, determine their location and cause, repair them,

retest them, and promote them again into the production environment. Finally, there is what may be the largest cost

of poor quality: business lost and the company’s reputation tainted by exposing end-users and customers to

systems, which cannot perform their functions accurately and dependably. This could result in inaccurate order

processing and billing, unexecuted transactions, miscalculations, inability of customers to do business during high

volume periods, generation of inaccurate reports, etc. Any one of these circumstances can cause customers to

discontinue their relationship with a company, and they may very well convince their acquaintances to do so as well.

The last major benefit to be derived from working with a vendor to establish a sound testing program is better

information to support management decision-making. Prioritizing requirements provides a sound basis for evaluating

product risk and determining areas for concentrating quality management or improvement programs. Measurements

of test completion and coverage represent necessary information for determining if and when products go live. Root

cause analysis of incident reports can lead to significant improvements in the product definition and development

process. A vendor can make recommendations for making the development effort more efficient and improving the

quality of its deliverables. It is this benefit, which is not immediately quantifiable in monetary terms, which can

contribute to the long-term success of the enterprise by helping senior management assess risk, set priorities,

establish product improvement programs, and generally provide a real-world basis for evaluating investments in

technology.

The bottom line? In some cases, development organizations prefer having their own test departments for reasons

that go beyond financial considerations. For one, they believe close coordination between test and development

resources requires proximity. In these instances, a hybrid solution of outsourced test specialists working on-site,

backed by external lab-based testing, may be the optimal solution. In either case, today's budget-minded and high

velocity organizations avoid missed time-to-market opportunities and enjoy lower over¬all costs of quality assurance

through testing outsourcing.

Whether you are a software product company and sell software or develop software to use internally, it is important

to begin with the discovery of business opportunities across the application lifecycle.

Choosing the testing processes that software companies want to outsource is a critical component of their

outsourcing initiatives and can have a tremendous impact on their success. The stakeholders should take the current

inventory of development and testing processes, resources, and tools before they start identifying objectives,

expectations, and metrics to measure the success of the company.

To forge a successful outsourcing strategy, companies should conduct an outsourcing ―readiness‖ assessment. The

assessment should be equally applied to companies considering outsourcing processes across the entire product

lifecycle, or to those who wish to outsource QA and testing only.

The results of the assessment will provide an organization with much of the information required to make key

decisions and to focus management effort. The assessment can help examine the ability of a software development

organization to outsource by looking at the processes maturity, experience, tools, and ROI objectives.

Data for assessment may be gathered from:

1. Questionnaires

2. Interviews

3. Discussion with product managers and application owners

4. Study of product roadmaps, test plans, test coverage, test scripts inventory, and application/code

base inventory

Deliverables may include:

1. Outsourcing Readiness Report

2. ROI analysis

3. Application/Project Sequencing Plan

4. Engagement Roadmap

When reviewing test plans, test coverage, and test scripts inventory, the following criteria are recommended to

assess readiness of specific processes:

1. Skill alignment with current/future quality assurance and testing direction

2. Current cost of testing/quality assurance

3. Application maturity

4. Alignment to business (core, context, utility)

5. Business value

6. Functional completeness

7. Technical completeness

8. Rate of change

9. Test coverage

10. Perceived risk profile

Successful completion of this phase will provide the company with insight to areas of the software quality assurance

and engineering, which have room for improvement. Gap analysis naturally flows from the results of the

Assessment/Business Case phase and involves determining, documenting, and approving the variance between

business objectives and current capabilities.

At this stage, companies should analyze gathered materials with the goal of comparing its actual performance with

its potential performance, and of finding gaps that represent opportunities for improvement.

The objective is to study the differences between the current and desired state/best practices of the quality

assurance and testing processes, for the purpose of determining how to get from one state to a new state. Gap

analysis is undertaken as a means of bridging that space. Among the various methodologies used to perform gap

analysis is IDEF, a group of methods used to create a model of a system, analyze the model, create a model of a

desired version of the system, and to aid in the transition from one to the other.

With that information in hand, a company can pinpoint and address the quality/process gaps. The ultimate goal is to

identify and compare a list of initiatives, which you can begin implementing to improve the speed, quality, and cost

parameters at each step of the product lifecycle.

In this step, companies define a high-level project plan and a test strategy for reaching the desired QA destination.

The following steps outline the major components of a typical outsourced QA project.

1. Overview –Includes the objectives for QA on this project and background information.

2. Scope – Contains what will be covered and what will not be covered in the QA effort.

3. Assumptions – Includes any assumptions made in developing the QA Strategy.

4. Project Organization – Contains the project organization chart and how the QA Team fits within.

Roles and responsibilities respective to QA and other stakeholders. Specify Project Team

responsibilities for test infrastructures and data.

5. Project Steps – Address how the QA effort will be approached. Include findings from Proof of

Concept. Include one or more schematics, along with written descriptions of the proposed QA

workflow. Describe types of data and validation methods.

6. Levels of Testing – Define what types of testing will be performed such as Unit, Integration,

System, Data Validation, Performance, GUI and the entrance and exit criteria for each.

7. Testing Groups - Testing groups provide a breakdown of major functionality. List the testing groups

and clearly state what they include and what they do not include.

8. Technical Assessment – Content depends on current status of development, technical

infrastructure, and type of technologies and platforms. In this section, the following should be done:

Assess compatibility of data structures along with any code or software specific issues. Address

feasibility of test automation with recommendations for using home-grown or vendor tools at certain

stages of the development cycle. Include recommendations regarding what should be automated for

testing.

9. Test Infrastructure Requirements - Assess existing infrastructure processes and needs. Discuss

test infrastructure considerations and requirements for loading converted data and ongoing data load

processes. Include environment sizing considerations and depth of test data. Clearly state how the

test infrastructure is and is not analogous to the production infrastructure. Note any exceptions or

limitations.

10. Processes and Tools – Describe processes and tools to be used for change control, configuration

management, and discrepancy reporting. Describe what test automation will or may be used and

how it will or could be performed.

11. High Level QA Project Plan - Schedule with tasks and associated resources for successfully

completing testing.

After implementation of the project, companies have to return to their initial assessment and gap analysis to compare

the results of the project with the initial expectations.

Test

Strategy

Risks

Impact of Failure

Likelihood of Failure

Application

Quality of Software

Nature of System

Strategic Significance

Test Tools

What to Automate

What Tool to Use

Manual Tools

Test

Objectives

Demonstrate Usability

Intrfaces

Coverage

Stress/Load

Test

Infrastructure

Code Control

Enviroments/Progression

Hardware

Software

Resources

Money

Time

People

Skills

Priorities

Hardware

Software

Test

Requirements

Completeness

Accuracy

Stability

Review

Strategy

Walkthroughs

Inspections

Desk Checks

During the execution phase the following metrics are used:

1. Planned test execution versus actual: are we on track? (could identify a risk for being on time)

2. Number of successful versus not successful tests (too low means inefficient testing due to low

quality of the test object)

3. Trend in outstanding number of defects (going up could mean a risk for being on time)

4. Burn rate of the test hours per phase (too high could identify a budget issue)

The metrics #1, #2, and #3 are essential to the success of an outsourced QA process and should be tracked weekly.

These metrics are mandatory for Quality Assurance projects with a duration of the execution phase longer than two

weeks.

By applying criteria to the metrics during a running test project, important signals can be derived as to whether the

outsourced testing project is going well and whether the quality of the test object is reasonable.

Towards completion of the project it is advisable to add the following metrics:

1. Has the test coverage reached sufficient level?

2. Is the percentage of successfully executed tests high enough?

3. Are the remaining defects below a certain level?

These metrics play an important role in the acceptance process.

When the execution phase is finished some additional metrics can be determined:

1. Number of hours spent on test execution per test case

2. Number of hours spent on test execution per defect found

3. Number of hours spent on testing as percentage of the whole project

When these metrics are collected for a number of comparable projects, they play an important role for supplier

management purposes (how efficient and effective is a test supplier) and in general to drive improvements.

When the project is over, it is recommended that companies analyze defects found after the deliverables were

accepted:

1. Number of defects found after acceptance compared to before acceptance

2. Test specification progress per week (number of test cases written) versus planned

3. Defect solution turn around time (are all showstoppers solved within a week?)

4. Defect root cause analysis in all phases (are there particular weak spots in build?)

The following mistakes are the most common when it comes to outsourcing Quality Assurance and testing activities:

1. No ownership of product quality.

2. No overall test program design or goals.

3. Non-existent or ill-defined test plans and cases.

4. Testing that focuses narrowly on functional cases.

5. NO ad hoc, stress, or boundary testing.

6. Use of inconsistent or incorrect testing methodology.

7. Relying on inexperienced testers.

8. Improper use of tools and automation, resulting in lost time and reduced ROI.

9. No meaningful metrics for tracking bugs and driving quality back into development.

10. Incomplete regression cycle before software release.

Future State, and Required Roadmap

Software vendors often make decisions about what and how to outsource too quickly, without detailed expert

assessment of their current state, desired future state, and required path on how to get there. It is critical to plan

outsourcing strategy by considering a company’s current software engineering processes maturity, in-house team

size, available documentation, and experience with distributed software development, cultural fit, and executive and

staff support. It is recommended to access applications, source code base, and business processes, and to lay out a

matrix of how and in what sequence applications and processes can and/or should be outsourced, kept in-house, or

retired.

The cost expectations that determine the savings that organization can leverage as a result of outsourcing are often

over-estimated. Although they can be based on ROI projections, software vendors should consider the learning

curve for the offsite team to become productive. For several years now, the business press has been claiming that IT

work costing $40-$80 an hour in the United States can be done for $15-$25 an hour in India or Russia. If those

figures sound too good to be true, that’s because they usually are. An offsite team often will not become productive

(by onshore standards) for at least three to four months, or even longer for more complex projects. IT executives

should expect to pay an additional 5 percent to 15 percent on managing an offsite outsourcing program, at least

during the first year. The transition phase will add costs, too — planned expenses should cover due diligence,

transfer of knowledge, and extensive program management.

Once a company has decided to outsource, it usually rushes to get the outsourcing deal done and put the

transaction on a fast track. To do that, executives from the company and the outsourcer meet, agree, and then let

the next level of management sort out all the details. Since an outsourcing process is a complicated one, it is

advisable that various elements of a company remain involved in the original meetings and have a say in the

agreement. Top-level executives alone are rarely sufficient to review such a process effectively.

The essence of outsourcing is that a company transfers the process of application engineering to the outsourcer, and

then buys the results of that process. The quickest and most certain way to destroy an outsourcing process is to

dictate to the outsourcer how the process should be done. When a company tells the outsourcer what to do, it

removes from the outsourcer the ability to add value to the process. As the result, the outsourcer cannot deliver

quality software.

Many companies choose not to transfer the ownership of the outsourcing process to the outsourcer, and instead opt

to keep the process under company control. When the company assigns someone to oversee the process, this

person usually becomes overwhelmed and is unable to manage it, because it is an area that he or she has very little

knowledge about. As a result the outsourcing process fails.

The longer your contract terms, the more deep you get in it. Information-driven enterprises and ASPs should

construct a contract that will last a long time, but that will allow the service term's agreements to change during short

periods of time. It is preferable for a company to develop a close relationship with an outsourcing company, but to

sign short-term contracts that can be easily negotiated and, if necessary, renegotiated or even broken.

Companies new to the outsourcing process often do not assign the right people to manage the process. They often

assume that since all the details of an outsourcing contract are discussed and the contract is signed, that they do not

have to worry about the outsourcing process any longer. The right person for the job is someone who understands

the process, is able to focus on soley on the process, and who can interpret the results with the best interest of the

company in mind. He or she should have a different way of thinking, in respect to most business managers, and the

oversight to recognize any problems or mishaps before they surface.

Far too often, the company forgets that the outsourcer is a business asset and must be treated as such. Team

members across the globe are vital parts of a successful and prosperous business. If the outsourcer is treated

differently— for example, like one of many suppliers or one of many customers—then the one thing that the

outsourcer adds to a company will be lost.

It is preferred that companies inspect in person the physical premises where the software is to be

developed. This is an opportunity to check the security firewalls of their vendor’s buildings and work

area, the organization culture, the functioning of their networks, etc.

It is preferable that companies interview the team members who will work on their project. This will

help companies to judge the employees' level of reliability.

Companies should also check the offshore development company’s employee retention rate, and

also determine if the outsourcing vendor is working with any competing organizations. If this is the

case, companies should ensure that the teams working on their competitors’ projects do not have

access to their project information.

Any method of information transfer, such as e-mails, fax, electronic file exchange, instant

messenger, on-line meetings, paper documentation etc, should have parameters for usage clearly

defined.

All activities that will have to accompany the end/termination of the contract should be defined while

negotiating the contract terms and conditions. This includes the retrieval of any methods and

procedures, documents, source and executable code, company proprietary security or development

standards, code libraries, and data stored offsite.

Contracts should be framed, such that the offshore company takes responsibility for the actions of its

employees.

Companies should ensure that any project-related work is not subcontracted without approval. This

will help to protect a sensitive application or data within the application.

Companies should ensure that the vendor agrees not to use any of the company’s confidential

information for purposes like sales, marketing, or demo without prior approval.

Companies should ensure that only data related to the performance and reliability of the system is

transmitted over the Internet. Information from the database should not be disturbed during any part

of the project.

Companies should make sure that the system experts make modifications to the system only after

obtaining prior permission.

Exchange of passwords and other critical information should be made secure by encrypting them.

Companies should ensure that the data used during testing does not expose the real information of

the customers.

Unwanted data should be destroyed.

Companies should ensure that the vendor reports on replacements of team members, if any.

Companies should ensure that an original copy of the source code is maintained.

The business benefits of including quality-oriented activities in all phases of your software development lifecycle are

both broad and deep. These measures not only facilitate innovation and lower costs by increasing predictability,

reducing risk, and eliminating rework, but they can also help to differentiate your business from its competitors. Most

importantly, continuously ensuring quality will always cost less than ignoring quality considerations.

Renat Khasanshyn is CEO of Altoros Systems, LLC. www.altoros.com

1. James M. Juran and Frank Gryna, Juran's Quality Control Handbook. McGraw-Hill, 1988

2. Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement, Jeff Tian

(Wiley-IEEE Computer Society Press, 2005)

3. Managing Successful IT Outsourcing Relationships, Petter Gottschalk and Hans Solli-saether (Jul

2005)

4. "Software Quality Assurance: Principles And Practice" by Nina S. Godbole Alpha Science

International, Ltd (2004)

5. Test Outsourcing: Risks and Rewards, 2006 Rich Yannetti 545 Fifth Avenue 4th Floor, Ste 401 New

York, NY 10017

6. Metrics for Outsourced Test Activities, Kees Blokland / Polteq IT Services B.V. (NL) 2004

7. Data Warehousing QA Challenges, Denise Copening, Olenick & Associates

8. http://en.wikipedia.org/wiki/Gap_analysis

9. Software Quality Assurance Improvement Plan: EPIcode Gap Analysis; May 2004, U.S. Department

of Energy

10. Facts and Fallacies of Software Engineering; Robert L. Glass; Addison-Wesley (2002)

11. See Watts S. Humphrey's article entitled "The Quality Attitude," published in the Software

Engineering Institute (SEI ) newsletter, (2004, no.3): http://www.sei.cmu.edu/news-at-

sei/columns/watts_new/watts-new.htm