Test defectmetricsanalysis080124

OAK~TQ Seminars 2008Celebrating 10 years of Quality Life

Test Defect Metrics and Analysis

Pradeep P. Oak

Bangalore[January 24, 2008]

Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/2

“Works are always associated with the defects .. just as the fire is enveloped with smoke”

Bhagavad-Gita Ch. XVIII, 48

2989/1B, 12th Main, 80 Feet Road, HAL II Stage,

BANGALORE – 560 008http://www.oaksys.net/


Who we are• Specialist Independent Software

QA/Testing company• Established in 1998• ISO 9001:2000 certified since 2003• Over 150 Test specialists• Multiple domain expertise• Offices in Bangalore, Pune• Operations in India, USA, Europe



Key Members • Mr. Pradeep Oak - M Tech., IIT Kanpur

– worked with TCS, Index, …– consulted with HPISO, Philips, DRDO, Satyam, …

• Mr. Narendranath Pai - B Tech – expertise in Business & Web Applications– worked with ANZIT, SAP Labs, PSI, …

• Ms Tejaswini - B Tech– expertise in Embedded, Real Time & Telecom projects– consulted with DRDO, Motorola, …

• Ms Asmita Oak - B Tech– expertise in Quality Management Systems– worked on V & V projects at DRDO, Synopsys, …

AND .. 150+ others


Services OfferedManaged Testing ServiceSoftware Test Planning & Strategy

developmentQA for the complete SDLCTest case design and driver developmentTest Environment setup & Test Execution Development of Test Automation FrameworkUser Acceptance Testing

In short, everything in Software Testing


What makes us different?• Flexible approach

To us, each client is unique. So is each client solution. • Adaptable delivery

We know that the key to a successful project is the ability to adapt. If anything is likely to have an impact on a project, we identify and communicate the issue quickly and take a constructive approach to problem resolution.

• Attention to detail• Superior technical ability• Team-based delivery

We don’t supply individuals - we supply the knowledge and backup of a specialist organisation. We have the depth to offer a full range of specialist skills, ranging from project management to automation scripting, focused on your requirements.

• Long term relationships, not "Touch and Go"


Our research initiatives• OAK-BVB Center of Excellence in V&V (2003)

• In association with BVB college of Engineering, Hubli

• Focusing on research and propoagation of new techniques in software Verification & Validation, deriving new methods of analysing test metrics, ..

• OAK-PDA Center of Excellence in Software Automation (2005)• In association with PDA College of Engineering, Gulbarga

• Focusing on test automation, tool evaluation, productivity improvement etc.


Projects’ Snapshot ..Testing & Test Automation in Insurance, Banking, CRM domainsSystem Testing, Automation and UAT of products under Insurance domain

• Cross-platform Testing of a Network Management Product• Testing WAP based internet software in Medical services domain

Testing and reverse engineering of a B2B software (Automotives)Test design, development and execution of ISDN v110 protocolInternet/Intranet based Pension administration softwareSystem Testing of Internet Payment Security SystemTest Automation of Telecom applicationsIV & V of of Mission Critical Avionics Software (LCA and others) under 2167A, DO-178B, IEEE 12207, …Test automation of telecom data validation systemRequirement analysis of Avionics system using formal methods (ESTEREL)UAT of HRMIS productManaged Testing of a Global Core Banking Application with CMMi L5 processes


Partial Client List• Aeronautical Development Agency, Min. of Defence, Govt. of India• ANZ Information Technology Ltd.• CEMILAC (Centre for Military Airworthiness & Certification)• CGI India• CyberCash India• Dell International Services Ltd• Electronics & Radar Development Establishment (LRDE)• First Apex Technologies• GE India Technology Center (GEITC) • IBM Global Services India Pvt. Ltd.• iFlex Solutions Ltd• India Life Hewitt Ltd. (leading Pension consulting ASP)• Infosys Technologies Limited • Price Waterhouse Coopers (PwC)• SupportSoft• Symantec (Veritas) India• Yodlee Inc.



Pradeep P. Oak



Seminar Agenda

• Tester – y’day – today - tomorrow• Defect metrics & Analysis

Discussions & experience-sharing of all• Questions & Comments


Where are we in Testing• 1980 .. Nothing called s/w tester• 1990 – QC/QA sneaks in• Late 90’s – Tester as a role• 2000’s – Independent testing• 2005 – Testing as a career option• many conferences, seminars,

trainings, symposia, ...• …• 2010 - ????. . . .


Industry ExpectationsIn the last few years .. We have seen lot of

changes ….Expectations are building up

• Need the right attitude• Act as Virtual Users• Job is quite demanding as the profile is

yet to get defined

. . . .


Tester is required to ..• Find all defects• Be responsible to own all the

onsite bugs• Know programming very well• Be a domain expert• Be analytical• Be flexible• Be available for a 9-6 job

(9 PM to 6 AM !!)• ….


Can we be a Thinking Tester ?

Goals• Be an efficient tester/test group• Improve effectiveness of testing• Do complete testing [goal accomplishment]• Be predictable [repeatable]

Let us try to discuss these thru sample metrics


Goal – Improve Test efficiency

Question: What is Test efficiency ?• Faster• Cheaper


Test Efficiency

Cost of finding a defect• No. of TCs designed v/s effort spent• No. of TCs executed v/s effort spent• No. of TCs automated v/s effort spent• Test effort v/s Devp effort• Waiting time v/s Test time


Cost of a defect

• Cost = effort spent/defects found• Avg. Effort per defect per project is plotted

Analysis..

Cost of a defect

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Pers

on H

ours


Cost of a defect - AnalysisWhy some projects have high costs of testing• Is it because of good quality of development?• Is it ineffective testing / inefficient testing ?• Is the product complex ?• Is it new technology ?• Is it new domain / platform ?• Is the team new ?• Is a different process followed / process not followed


Test Effort distribution

0

5

10

15

20

25

30

35

TestStrategy

STP ITP UTP UT IT ST

Testing Activity

% T

est E

ffor

t

P1P2P3

Analysis..


Test Effort distribution - Analysis

• Is there a relationship between planning & execution ?• Is more planning leading to reduced execution time

(telescopic)?STP v/s STITP v/s ITUTP v/s UT

Related Question: Is there a relationship between test effort & test effectiveness


Cost v/s Value of a Test

• What is the COST of each test ?• What is the VALUE of each test ?


Cost v/s Value of a Test• When you test what you gain ?

• When you do not test, what you save/gain ?

• Cost benefit analysis or ROI– Tangible benefits– Intangible benefits –

• increased confidence, better image

Why not .. Value of Quality (VoQ) instead of CoQ !!


Goal – Improve effectiveness

Question: What is Test effectiveness ?• Finding all defects [till current stage]• No defects reported in next stages• Finding critical defects more • Presenting Testing work well to others


Goal – Improve effectiveness

What are the metrics ?• Defect leakage• Defect distribution• Defect prediction• Defect acceptability by affected teams• Feedback / Survey• …..


Defect Leakage Matrix (sample)SRS Dsgn CUT IT ST UAT Total

SRS 23 5 1 8 2 39Dsgn -- 56 7 1 64CUT -- -- 234 6 34 4 278

IT -- -- -- 32 3 2 37ST -- -- -- -- 128 26 154

UAT -- -- -- -- -- 11 11Total 23 61 242 39 173 45 583

Def

ect i

nduc

tion

Defect detection (reviews/testing) Analysis..


Defect Leakage Matrix - Analysis• Ideal matrix should have values only on the diagonal cells• Any values above the diagonal indicate defect leakage

Your Comments on defect leakage please ?? ..

• How is the testing process ? UAT / ST / IT / UT


Defect classificationDefects by Severity

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

Showstopper High Medium Low Misc

Severity

% d

efec

ts

Analysis..


Defect classification - AnalysisAre important defects being identified ?

It is also important to look at ..Who is assigning severity ?Important in whose eyes ?

Are defect classification guidelines defined and agreed


Defect Prediction ModelDefect distribution across SDLC

11

13.5

16.5 17.25

10.5

21

10.25

7

17

0

5

10

15

20

25

SRS Design Coding UT IT ST UAT

Phase

% D

efec

ts

Avg %

No. of defects NOW

U R here

UAT

Analysis..


Defect Prediction Model - AnalysisFrom past experience, 10% defects got skipped to UAT

Suppose, We have committed to deliver with less than 10 defects at UAT.

And, found 7 defects at SRS & 17 defects at Design• What do we infer ?• What is expected at UAT ?• Quality of SRS development v/s quality of SRS review• Quality of Design v/s quality of design reviewSuppose, Till the time of delivery, we had already found 400

defects.• What do we predict at UAT ?


Defect Acceptability

0

50

100

150

200

250

# of

def

ects

Code Rev UT IT STPhase

TotalDefectsAcceptedRejectedWithdrawn

Analysis..


Defect Acceptability - AnalysisDefects acceptedDefects rejectedProbable Defects withdrawn

• Comments please ..

• How early defects are reported ? Is there enough time to fix them?

• Tester’s Image & Pride are at stake


Goal –Test completeness

Question: What is Test completeness ?• Functional & nonFunctional• Against standards / objectives


Goal Accomplishment (FURPS+)

0%10%20%30%40%50%60%70%80%90%

100%

Functi

onali

ty

Usabilit

y

Reliab

ility

Perfor

mance

Secu

rity

Main

taina

bility

Supp

ortab

ility

TC not ApplTC not ExecTC FailedTC Passed

Analysis..


Goal Accomplishment - Analysis

Test cases failed & not executed are important to look at

Is there a need to replan ?


Completeness / Coverage

Feature coverageCode coverage – statement, condition, loop, ..Requirements coverageStandards/protocol coverage


Goal – Create right visibility

• Planned presentations to management– what– How often

• Conduct survey among affected groups • Collect feedback – internal / external


Justify OurselvesDevelopers feel-“ Testers write lot of tests, but it is a wasted

effort. Real bugs are not uncovered.”Is this true?– May be!– May be not!!

How do we justify?Measurement - Metrics - Analysis


Testers must …• Know what they are doing• Have pride in their work• Present their work to the right people

• Plan, Measure, Analyse, Refine, ….

Industry responds with rewardsIt is an opportunity to be a leader


Going Forward ...• Industry will learn & mature• Demand & value of Testing is

bound to increase• Will have matured models to support• Focus may then shift more

towards preventive from corrective

Thank YouYour feedback is valuable to us

- We can be contacted at -OAK SYSTEMS PVT. LTD.

http://www.oaksys.net/[email protected]

Phone- (91)-(80)-4126 7681


Q & CQuestions and Comments



Pradeep P. Oak
