Coverage Discounting: Improved Testbench Qualification by
Combining Mutation Analysis with Functional Coverage Nicole
Lesperance, Peter Lisherness, and Tim Cheng Univ. of California,
Santa Barbara September 9th, 2013 1
Slide 2
Verification Questions 1.Have the test vectors adequately
stimulated the design? 2.Is the testbench able to identify errors?
2
Slide 3
Motivation 3 Functional Coverage Mutation Analysis Coverage
Discounting + Functionally Meaningful + Evaluates design activation
-Ignores propagation and checker quality + Evaluates propagation
and checker quality - Mutants hard to analyze + Functionally
Meaningful + Evaluates design activation, propagation and checkers
Functional Coverage Mutation Analysis Coverage Discounting
Slide 4
Coverage Discounting: The Main Idea Use fault insertion to
discount (revise downward) coverage scores 4 Detected + undetected
mutants Functional Coverage Mutation Analysis Coverage Discounting
Undetected Mutants Coverage Score 100% 0% Discounted Coverage
Score
Slide 5
An Existing Solution: Mutation Analysis DUT Tests Checker
Undetected Fix Add/Fix Detected Benefits Evaluates quality of
testbench checkers Indicates if tests fail to propagate potential
errors to the checker or fail to activate mutants 5
Slide 6
Mutation Analysis Drawbacks 1.Long runtime for simulation
2.Many man-hours required to analyze results: 2.1 Mutants are
synthetic - difficult to identify testbench improvements to target
mutant detection 2.2Redundant mutants never detectable: Some
undetected faults are redundant if (a > 0) b=1+a; else b=1-a; if
(a >= 0) b=1+a; else b=1-a; Example: 6
Slide 7
Premise of Coverage Discounting 1) A mutant may change design
functionality 2) A coverpoint, covered by testbench in the original
design, may be suppressed (i.e. no longer be covered) in the
mutated design 3) If the mutant suppressing the coverpoint is not
detected, then the coverpoint is no longer considered covered in
the original design 7
Slide 8
Coverage Discounting Flow 8 DUT Tests Checker Detected
Undetected Add more tests No Yes Fix Coverage Changed?
Slide 9
Experimental Results 9
Slide 10
OpenRISC CPU Experiment #1: Measure test quality Tests:
Instruction-Set v. Random Coverpoints: Opcodes Experiment #2:
Identify checker weaknesses Tests: SoC Regression Suite
Coverpoints: CPU top-level signals 10
Slide 11
Measure Test Quality Functional Tests Original Discounted
Random Tests 11
Slide 12
Identify Checker Weaknesses Tests 12
Slide 13
Identify Checker Weaknesses 13
Slide 14
UART Illustrates success of coverage discounting applied to a
sophisticated testbench Simulation Infrastructure: Design RTL w/OVM
Testbench SystemVerilog coverpoints (ModelSim) Certitude fault
insertion Tests: OVM Sequences Coverpoints: Hand-written spec-based
14
Slide 15
UART Results Reduces debug effort from analysis of 146 mutants
to examining 3 functional coverpoints 1588 Mutants: 7 not
activated, 106 not propagated, 33 not detected Total 146 mutants
demand attention 846 Coverpoints: 4 uncovered 3 discounted All
discounted relate to specific unchecked functions Loopback, timeout
interrupt identification register 15
Slide 16
Results Examined Coverage discounting: Identified "good" tests
(propagation) Diagnosed checker problems Identified coverpoints
vacuously covered However: Runtime of mutation analysis still an
issue Technique sensitive to the quantity and quality of faults
16
Slide 17
Confidence Metric Aims to answer: 1.Is a coverpoint adequately
challenged by the set of mutants? 2.When have enough faults been
inserted (when can simulation be stopped)? 3.What is the optimal
simulation ordering of mutants for coverage discounting? 17
Slide 18
Quicker Confidence CPUUART 18
Slide 19
Quicker Discounting CPUUART 19
Slide 20
Technology Transfer Working closely with Freescale to apply
discounting technique to industry design Jayanta Bhadra and
Vivekananda Vedula Publications Lisherness, P., Lesperance, N., and
K. T. Cheng, Mutation Analysis with Coverage Discounting, Design
Automation and Test Europe (DATE), March 2013. Lisherness, P., and
K-T. Cheng, "Improving Validation Coverage Metrics to Account for
Limited Observability", Asia and South Pacific Design Automation
Conference, Feb. 2012. 20
Slide 21
Summary Analyzed information gained from coverage discounting
for two designs Developed a confidence metric to gauge mutant
effectiveness and an ordering heuristic to reduce runtime 21