05 26 2011 Metrics for Testing

7/31/2019 05 26 2011 Metrics for Testing

1/20
http://www.rbcs-us.com/


2/20

Metrics for Testing: Managing with Facts

By using metrics, we can manage withfacts and reality

How can we use metrics to manage testing?

What metrics can we use to measure the testprocess?

What metrics can we use to measure ourprogress in testing a project?

What do metrics tell us about the quality ofthe product?

Lets see if we can find some answers

Metrics for Testingwww.rbcs-us.com

Copyright (c) RBCS 2005-2011 Page 2
http://www.rbcs-us.com/http://www.rbcs-us.com/


3/20

Why Metrics?

Managing with metricsfor testing oranythingallows us to manage withfacts

Subjective, uninformed opinions are nota sound basis for management

What sounds reasonable can be wrong(e.g., Aristotle)

Metrics for Testing Page 3www.rbcs-us.com

Copyright (c) RBCS 2005-2011


4/20

Why Testing Metrics?

Testing by itself has no valuebut it producespotentially valuable information

Test information must be generated and

communicated effectively to be valuableCommon communication goalsNotify (We have 24 bugs remaining to close)

Enlighten (See the time lost due to reopens)

Influence (We propose a bug triage meeting)

The metrics should be used as part of regularstatus reports (dashboards) as well as periodicanalyses




5/20

How to Develop Metrics?

Identify objectivesCreate questions about effectiveness,efficiency, and elegance of achievement of

objectivesDevise a measurable metric (or find asurrogate metric) to address the questionSet a goal for each metric

Implement improvements to improve goalLets look at types of metrics andexamples




6/20

Example

Objective: Find defects

Effectiveness question: Have we

finished finding new defects?Metric: Track trend in defect discoveryduring testing

Goal: Watch for convergence (morelater)

Improvement: Find bugs earlier


http://www.rbcs-us.com/http://www.rbcs-us.com/http://www.rbcs-us.com/


7/20

Example

Objective: Build confidenceNote that we need a surrogate metric forconfidence; one way is to measure coverageCoverage has multiple dimensions

Effectiveness question: Do any requirements haveidentified failures?Metric: What % of requirements are completelytested without any failures, what % requirementshave failures, what % are untestedGoal: 100% requirements tested without must-fixfailuresImprovement: Reduce % of requirements that failin testing




8/20Metrics for Testing Page 8

Good Metrics

Define a useful, pertinent, and conciseset of quality and test metrics

Avoid too large a set of metricsToo difficult to measure later

Confusing to participants

Ensure uniform, agreed interpretations

of these metrics, to minimize disputesand divergent opinions about measuresof outcomes, analyses, and trends

www.rbcs-us.comCopyright (c) RBCS 2005-2011


9/20Page 9

Tracking Metrics

Automated tool support can reduce the time requiredto capture, track, analyze, report, and measuremetrics

Apply objective and subjective analysis for specificmetrics over time, especially when trends emergethat could allow for multiple interpretations ofmeaning

Be aware of and manage the tendency for peoplesinterests to affect the interpretation they place on aparticular metric or measure




10/20Page 10

Reporting Metrics and Measurements

Reporting of metrics and measures should enlightenmanagement and other stakeholders, not confuse ormisdirect them

Good testing reports based on metrics should be

easily understood, not overly complex or ambiguousGood testing reports based on metrics should helpmanagement improve processes, guide the project tosuccess, and manage product quality

Metrics include snapshots of status at a moment intime, trends emerging over time, and analysis ofcauses and relationships between factors thatinfluence testing and quality outcomes




11/20

Process Metrics for Testing

UsesTo measure the effectiveness, efficiency, andelegance of the test processTo provide the insights to guide whereprocess improvements should occur

Part of certain test process assessments(e.g., using CTP framework)Not used as frequently as should be thecaseNot typically part of any dashboard (butcan be)




12/20

Developing Good Process Metrics

Determine key objectives for the testprocessTypical objectives

Finding bugs, especially important bugsReducing quality risk

Decide which important effectiveness,efficiency, and elegance questions apply

Create metrics that address thosequestionsSet goals that reflect current or desiredcapability




13/20

Bug Finding Effectiveness

Objective: Finding bugsQuestion: What percentage of bugs found?Metric: Defect detection percentage

Goal: A typical number is 85%, thoughsome teams do much better

Metrics for Testing Page 13

presentbugs

detectedbugs=DDP

bugsproductionbugstest

bugstest)(

+

=testingforDDP

www.rbcs-us.comCopyright (c) RBCS 2005-2011


14/20

Bug Finding Focus

Objective: Finding important bugs

Question: Do we find more important

bugs than less important bugs?Metric: Relation of DDPs (see below)

Goal: Relationship shown below is

always true

Metrics for Testing Page 14

)()( bugscriticalDDPbugsallDDP

Documents

05 26 2011 Metrics for Testing