Testing Metrics: Project, Product, Process

Metrics for Software TestingMeasuring Project, Process, and

Product

Metrics for Testing: Managing with Facts

By using metrics, we can manage with facts and reality

How can we use metrics to manage testing? What metrics can we use to measure the test process? What metrics can we use to measure our progress in testing a project? What do metrics tell us about the quality of the product?

Let’s see if we can find some answers…Metrics for Testing

www.rbcs-us.com Copyright (c) RBCS 2005-2012 Page 2

Part 1: Why and How of Metrics

Why Metrics?

Managing with metrics—for testing or anything—allows us to manage with factsSubjective, uninformed opinions are not a sound basis for managementWhat sounds reasonable can be wrong (e.g., Aristotle)

Metrics for Testing Page 4www.rbcs-us.com

Copyright (c) RBCS 2005-2012

Why Testing Metrics?Testing by itself has no value…but it produces potentially valuable informationTest information must be generated and communicated effectively to be valuableCommon communication goals

Notify (“We have 24 bugs remaining to close”)Enlighten (“See the time lost due to reopens”)Influence (“We propose a bug triage meeting”)

The metrics should be used as part of regular status reports (dashboards) as well as periodic analyses



How to Develop Metrics?Identify objectives Create questions about effectiveness, efficiency, and elegance of achievement of objectives Devise a measurable metric (or find a surrogate metric) to address the question Set a goal for each metricImplement improvements to enhance achievement of objectivesLet’s look at two examples…



Example One

Objective: Find defectsEffectiveness question: Have we finished finding new defects?Metric: Track trend in defect discovery during testingGoal: Watch for convergence (more later)Improvement: Find bugs earlier



Example TwoObjective: Build confidenceNote that we need a surrogate metric for confidence; one way is to measure coverageCoverage has multiple dimensionsEffectiveness question: Do any requirements have identified failures?Metric: What % of requirements are completely tested without any failures, what % requirements have failures, what % are untestedGoal: 100% requirements tested without must-fix failuresImprovement: Reduce % of requirements that fail in testing



Metrics for Testing Page 9

Good MetricsDefine a useful, pertinent, and concise set of quality and test metrics Avoid too large a set of metrics

Too difficult to measure later Confusing to participants

Ensure uniform, agreed interpretations of these metrics, to minimize disputes and divergent opinions about measures of outcomes, analyses, and trendswww.rbcs-us.com


Page 10

Tracking MetricsAutomated tool support can reduce the time required to capture, track, analyze, report, and measure metrics Apply objective and subjective analysis for specific metrics over time, especially when trends emerge that could allow for multiple interpretations of meaningBe aware of and manage the tendency for people’s interests to affect the interpretation they place on a particular metric or measure

Metrics for Testingwww.rbcs-us.com


Page 11

Reporting Metrics and Measurements

Reporting of metrics and measures should enlighten management and other stakeholders, not confuse or misdirect themGood testing reports based on metrics should be easily understood, not overly complex or ambiguousGood testing reports based on metrics should help management improve processes, guide the project to success, and manage product qualityMetrics include snapshots of status at a moment in time, trends emerging over time, and analysis of causes and relationships between factors that influence testing and quality outcomes



Part 1: Why and How of Metrics

Exercise

Develop a MetricPick any personal, professional, or business objectivePose a question about the effectiveness, efficiency, and elegance of achievement of this objective Devise a measurable metric (or find a surrogate metric) to address the question Set a goal for each metric



An Example

Objective: train for a marathonEffectiveness question: Can I complete the marathon?Metric: starting with a 10 km run, run a long run every weekend, adding 2 km every weekendGoal: Continue for 14 weeks until reaching 38 km two weeks before actual marathon



Part 2: Process Metrics

Process Metrics for TestingUses

To measure the effectiveness, efficiency, and elegance of the test processTo provide the insights to guide where process improvements should occur

Part of certain test process assessments (e.g., using CTP framework)Not used as frequently as should be the caseNot typically part of any dashboard (but can be)



Developing Good Process Metrics

Determine key objectives for the test processTypical objectives

Finding bugs, especially important bugsReducing quality risk

Decide which important effectiveness, efficiency, and elegance questions applyCreate metrics that address those questionsSet goals that reflect current or desired capability



Bug Finding EffectivenessObjective: Finding bugsQuestion: What percentage of bugs found?Metric: Defect detection percentageGoal: A typical number is 85%, though some teams do much better


present bugs

detected bugsDDP

bugs production bugstest

bugstest )(

testingforDDP

www.rbcs-us.com Copyright (c) RBCS 2005-2012

Bug Finding Focus

Objective: Finding important bugs Question: Do we find more important bugs than less important bugs?Metric: Relation of DDPs (see below)Goal: Relationship shown below is always true


)()( bugscriticalDDPbugsallDDP www.rbcs-us.com






Part 2: Process Metrics

Exercise

Test Process Metric

Define an objective for your test processDefine an effectiveness, efficiency, or elegance questionCreate a metric that addresses that questionsDiscussMeasure your own test process when you get back to your office



Part 3: Project Metrics

Project Metrics for TestingUses

To measure the progress of the testing effort on a projectTo provide the insights to guide where project optimizations might be needed/useful

Part of certain test process assessments (e.g., using CTP framework)The most commonly used test metricsUsually included in test dashboards (often over-relied upon)













Page 31

A Troubled Project’s Defect Metrics



Part 3: Project Metrics

Exercise

Your Current Test Project Metrics

Sketch or describe your current test project metrics Classify your presentation of each metric as:

Snapshot of status (e.g., tables)Trend chartAnalysis/causality

What do you think is missing from your metrics?Discuss



Part 4: Product Metrics

Product Metrics for TestingUses

To measure the quality and residual quality risk in the productTo provide the insights to guide where product improvements should occur

Good testing should measure the quality of the systemNot always measured during testing (using project metrics instead)Not always on testing dashboards (but should be)



Page 36

Example: Reporting Residual Levels of Risk

Test execution period

Region in green represents risks for which all tests were run and passed and no must-fix bugs were found. Region in red represents risks for which at least one test has failed and at least one must-fix bug is known. Region in black represents other risks, which have no known must-fix bugs but still have tests pending to run.



Page 37

Example: Testing and RiskDefects Tests

Quality Risk Areas # % Planned Executed

%

Performance 304 27 3,843 1,512 39

Security 234 21 1,032 432 42

Functionality 224 20 4,744 2,043 43

Usability 160 14 498 318 64

Interfaces 93 8 193 153 79

Compatibility 71 6 1,787 939 53

Other 21 2 0 0 0

Total 1,107

100 12,857 5,703 44



Use of Standards


Functionality: Suitability, accuracy, interoperability, security, compliance

Reliability: Maturity (robustness), fault-tolerance, recoverability, compliance

Usability: Understandability, learnability, operability, attractiveness, compliance

Efficiency: Time behavior, resource utilization, compliance

Maintainability: Analyzability, changeability, stability, testability, compliance

Portability: Adaptability, installability, co-existence, replaceability, compliance

Addressed by functional tests

Addressed by non-functional tests




Part 4: Product Metrics

Exercise

Your Current Product MetricsSketch or describe how you currently measure the product during testing

Coverage of requirements, configurations, etc.Residual level of riskConfidence in the product

Sketch or describe how your organization currently measures product quality during actual useHow well do your pre-release metrics predict product quality as experienced by the users and customers?Discuss



ConclusionsMetrics can be useful to measure, understand, and manage process, project, and product attributesTesting metrics are essential for a complete understanding of statusUse a small number of metricsUse surrogate metrics where necessaryEnsure consistent stakeholder understanding of metricsDefine objectives, then questions, then metrics, then goalsUse the goals to drive meaningful process improvements




For almost twenty years, RBCS has delivered consulting, outsourcing and training services to organizations that want to improve their software and hardware testing. Employing the industry’s most experienced and recognized consultants, RBCS assesses test teams, manages testing projects, builds and improves testing groups, and provides test experts for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved system development, decreased failures in production, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.Address: RBCS, Inc.

31520 Beck RoadBulverde, TX 78163-3911USA

Phone: +1 (830) 438-4830E-mail: [email protected]: www.rbcs-us.com

To Contact RBCS


Technology

Testing Metrics: Project, Product, Process