Prioritizing Test Cases for Regression Testing Based on Boundary Value Coverage

Tao Xie and David NotkinComputer Science & Engineering, University of Washington

IntroductionProblem: If resources are constrained, how can we still perform regression testing effectively?

Solutions: Test Case Selection selects a subset of test cases that exercise those changed entities

Test Case Prioritization orders the test cases with the intent of executing the test cases most likely to expose faults earlier

Structural coverage: exercising structural entities (e.g., statements, edges) that encode faults is essential to exposing those faults try to choose test cases that achieve high coverage of those entities.

Data Coverage: exercising entities with special values (e.g., boundary values) increases the ability to expose many faults try to choose special values that achieve high coverage of those entities.

What test cases are most likely to expose faults ?

What special values are dangerous?Boundary values tend to be dangerous. Extensively discussed in practitioner and researcher's testing handbooks: “The Art of Software Testing” by G. J. Meyers, 1979 “Software Testing Techniques” by B. Beizer, 1990 …… and research literatures ….

Related Work:Boundary Value Coverage

Previous empirical studies showed data coverage techniques, e.g. boundary value coverage, outperformed structural coverage ones in testing effectiveness

Complementing structural coverage techniques with data coverage ones are suggested in practice.

Related Work: Test Case PrioritizationMost existing test case prioritization techniques are based on

structural coverage.

Invariants

Instrumented program Original

program

Test suite

Run Instrument Data trace database

Detect invariants

Extract BP

Boundary Predicates

Calculate BVC

BVC

Most existing applications of boundary value coverage are in test case generation.

is based on Usually the specification or an assertion-like annotation is required to infer boundary value. Few programs are equipped with them.

Lack of tool support to collect boundary value coverage information without a priori specification.

Supportive Evidence

Challenging Problem

Our Solution: Test Case Prioritization Based on Boundary Value Coverage

without a priori specificationBoundary predicates (used to infer boundary values) are extracted from invariants dynamically detected by Daikon, an invariant detection tool. The boundary value coverage for a test case execution is collected based on the data trace of that execution and the boundary predicates. Then test case prioritization can incorporate this boundary value coverage information to improve performance.

This work was supported in part by the National Science Foundation under grant ITR 0086003.

Standard characterizations of test cases include:

Our initial experiments show promise for boundary value coverage; in particular, boundary value coverage dominated structural coverage for these small programs.

Average of the Precentage of Faults Detected (APFD) for 7 Siemens programs ("testing benchmarks")

APFD Values range from 0% to 100% and higher values imply faster or bettter fault detection rate.

0.78

0.8

0.82

0.84

0.86

0.88

0.9

0.92

1 2 3

Test Case Prioritization Techniques

APF

D

APFD

Boundary Value Coverage

Function Coverage Random