Upload
chiportal
View
1.021
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
May 1, 2013
Metric Driven Formal Verification
Raik Brinkmann
OneSpin Solutions
May 1, 2013
OneSpin Solutions - Who we are…
OneSpin provides enduring solutions that
enable the most thorough & easiest to use logic verification available
OneSpin Solutions was spun out of Infineon in 2005
Technologies include Assertion-based
verification (ABV) and equivalence checking (EC)
Production-proven tools used in
thousands of designs
The company is a completely independent entity owned by
Azini Capital and OneSpin’s employees
The original technologies have been augmented & enhanced to provide the most thorough formal capabilities available
Engineering HQ based in Munich, Germany
Sales offices in US, France, Germany, Israel, Japan, and
Turkey
May 1, 2013
Formal Verification with OneSpin 360
• Unique push button observation coverage analysis for verification progress
• Faster assertion development in timing diagram style using transaction level assertions
• 100% functional coverage using unique gap free verification
• Unique structural assertion debugger and active value/driver tracing through RTL
• Incremental compilation of assertions for quick turn around
• Automatic clock and reset detection for easy design setup
Unique productivity features
• HDL-linting
• Structural assertion synthesis
• Coverage closure for simulation
• SoC connectivity verification
• Register verification
• Formal score boarding
• Protocol verification IP
• X-propagation verification
Large selection of push-button design verification solutions, included
• Verification of logic, sequential and power optimizations
• Supporting RTL and gate level for FPGA and ASIC
• Integrated with major vendors and many synthesis flows
Push-button equivalence checking
May 1, 2013
Outline Importance of Progress Metrics
Coverage Metrics
Control vs. Observation Coverage
Practical Observation Coverage
Combining Control and Observation Coverage
Examples
Summary
May 1, 2013
Why do we need a metric?
DUV
Tests / Scenarios Checkers
Bus Functional
Model (BFM)
Test #1
Test #2
Stimulus / Scenario Generation
Constraints Formal
Engine
Stimulus / Scenario Exclusion
Check #1
Check #2
Check #3
Check #4
Simulation
Formal
Verification Environment (simplified):
Verification Process:
• Plan: Verification planning
• Do: Write tests / properties and verify and fix DUV, tests, properties
• Check: Are we done, (still) making progress?
• Act: Adapt planning or get ready for tape-out
Progress Metric : Key Component of Verification Flow
Plan Do
Check Act
May 1, 2013
Standard Formal ABV
Progress of Formal Verification?
• Unclear how many assertions to write and where to put them
• Unclear verification progress
• Unclear how formal verification affects overall verification quality
Integration with Simulation?
• Weak integration into verification flow and sign-off
• Additional effort to existing simulation effort
• Formal used as “point tool” for verifying specific aspects (e.g. hard to test corner cases)
Low acceptance of formal ABV
We need a practical progress metric!
May 1, 2013
Progress Metric & Coverage Taxonomy
Coverage
Structural Functional
Progress Metric
Bug Rate
Code Circuit
• Line/Statement/Block • Expression/Branch/Condition • FSM
• Toggle • Latch
Scenarios
Implementation artefacts
Analytical
Bugs Fixed
100%
time
ok #
Assertion Complete
• Gap Free Analysis • Verification Plan
Anecdotal
Have I written enough tests / properties? How much has been verified? Where are the gaps in my verification? Are we done and ready for tape-out?
if (…)
…
if (…)
else …
May 1, 2013
Control / Simulation Coverage:
• Focused on quality of stimuli
• How to judge quality of checkers?
Observation & Control Coverage
coverage metrics
stimulus generation checkers/assertions
plan
DUV
report
How good are my test vectors & constraints?
How good are my checkers and assertions?
How much of my DUV is verified? When is my verification finished?
Observation Coverage:
• Focused on quality of checkers
• Exposes unverified DUV parts
May 1, 2013
Been there? Done what?
• Has the statement been reached?
• Idea:
– If a statement has not been reached during verification, it can’t break a check.
– If a statement has been reached, would some check fail?
• Can measure quality of stimuli.
case (state)
…
burst:
if (cancel_i)
done_o <= 1
…
active
case (state)
…
burst:
if (cancel_i)
done_o <= X
…
mutate
• Has the statement been verified?
• Idea:
If a statement is modified, some check should fail.
But would some check fail, if the statement cannot be reached?
• Can measure quality of checkers.
Been there! Done that!
Example: Statement Coverage
Coverage
Control Observation
Been there! Done that!
May 1, 2013
Control Coverage for Formal No explicit stimulus generation for formal
Formal considers all possible input stimuli by default (exhaustive verification)
However: Control coverage still an issue because
• Formal requires constraints excluding illegal environment behavior
• Constraints may accidentally be too strong and exclude vital functionality
• Constrained functionality can neither be controlled nor observed
(Structural) control coverage quantifies degree to which locations of a design have been activated during verification.
control
controllable
reached
unreached
unknown
uncontrollable
dead
constrained
May 1, 2013
(Formal) Observation Coverage Verification tries to answer the question:
• “Does a design satisfy a specification?”
(Observation) coverage tries to answer the question:
• “What causes a design to satisfy a specification?”
Causality:
• “When do we say A is a cause of B?”
Common approach: Counterfactual causality:
• “A is a cause of B if, had A not happened, then B would not have happened.”
• A code line is a cause of a design satisfying an assertion. Source: H. Chockler, IBM
(Structural) observation coverage quantifies degree to which locations of a design have been responsible to make checks pass.
observation
unknown
observable
observed
unobserved
unobservable
May 1, 2013
Practical Observation Coverage
Mutation coverage
• Particular changes depending on error model, usually static
• Replacing for example “a <= b;” with “a <= 1´b1;”
• Problems:
• Can lead to vacuously holding assertions
• Requires several modifications at each location, high run time
“Quantify MDV” metric
• Use abstraction by free variables, allow dynamic behavioral change
• Replacing for example “a <= b;” with “a <= free_b;”
• Advantages:
• No unintended vacuity
• One modification for each location leads to faster results
• Multiple locations can be checked at the same time to prove “unobserved”
May 1, 2013
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= b;
3'b100: z <= c;
default: z <= <input>;
endcase
end
M5
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= b;
3'b100: z <= <input>;
default: z <= 1'b1;
endcase
end
M4
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= <input>;
3'b100: z <= c;
default: z <= 1'b1;
endcase
end
M3
Formal Observation Coverage Example module select1(onehot, a, b, c, z, clk, reset); input clk; input reset; input [2:0] i; input a; input b; input c; output reg z; always @(posedge clk or posedge reset) if (reset) z <= 1'b0; // L1: not covered (reset case) else begin case (i) 3'b001: z <= a; // L2: covered by assertion 3'b010: z <= b; // L3: not covered 3'b100: z <= c; // L4: not covered default: z <= 1'b1; // L5: not covered endcase end // if there is no reset, then 'a' is stored in 'z' if ‘i' is 3'b001 A: assert property ( @(posedge clk) disable iff (reset) i == 3'b001 |=> z == $past(a) ); endmodule
Q: Which assignment locations Lx in design M are observed by proven assertion A?
2. Re-Check property A for each M1..M5
always @(posedge clk or posedge reset)
if (reset)
z <= <input>;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= b;
3'b100: z <= c;
default: z <= 1'b1;
endcase
end
M1
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= <input>;
3'b010: z <= b;
3'b100: z <= c;
default: z <= 1'b1;
endcase
end
M2
Assertion A holds on M1: L1 not observed
Assertion A fails on M2: L2 is observed
M
A
L3 not observed
L4 not observed
L5 not observed
1. Modify each location L1..L5
of M: Producing M1..M5
A: The locations Lx for which A fails after replacing the assignment with a free input.
May 1, 2013
Merging Observation and Control Observation Coverage Control Coverage Merged Result
observed reached1) covered
unknown reached reached
unobserved reached unobserved
unknown unreached unreached
unobserved unreached uncovered
unobservable2) constrained constrained
unobservable2) dead dead
1. If a location is uncontrollable, it is also unobservable. 2. If a location is observed, it is also reached and thus controllable.
observation
unknown
observable
observed
unobserved
unobservable
control
controllable
reached
unreached
unknown
uncontrollable
dead
constrained 2.
1.
Been there! Done that!
May 1, 2013
Interfacing with UCIS
UCIS
• Coverage standard evolved from Mentor Graphics‘ UCDB
• Only supports control coverage, but not observation coverage
How to interface?
• Use user defined structures to store Quantify MDV results
• Use result merging to enhance standard UCIS coverage
Result Merging from UCIS to Quantify MDV:
• Covered bins -> Reached locations
Merging Quantify MDV to UCIS:
• Dead/Constrained -> Coverage Excludes
• Reached locations -> Covered bins
• Covered locations -> Covered bins
Conclusions on UCIS interfacing
• Merging results with UCIS is possible and useful now
• Direct support for observation coverage by UCIS is desirable
May 1, 2013
Simple Quantify MDV Example • Mixed control and observation
coverage results indicate:
– holes indicating missing assertions
– constrained code indicating over-constraining verification
hole
verified code
constrained code
dead code
Management overview
May 1, 2013
Industry Examples
Design LOCs #Assertions #Locations Runtime
FIFO 321 30 21 100s
FSM-DDR2-Read 839 6 93 106s
vCore-Processor 295 8 86 204s
SQRT 383 2 35 257s
IFX-Aurix-1*) 25563 85 2316 4d
IFX-Aurix-2 27374 157 1993 5d
IFX-Aurix-3 57253 253 5309 7d**)
*) “Quantification of Formal Properties for Productive Automotive Microcontroller Verification”, Holger Busch, DVCon, San Jose - February 26, 2013 **) interrupted at 80% completion
May 1, 2013
Summary
Coverage is an important aspect of verification
New metric introduced: “Quantify MDV”
• Combines observation and control coverage
• Agnostic to assertion style, assertion language, design style and design language
• Scales to relevant big industry designs
• Useful for
• Quick feedback on formal verification quality, as well as
• Criterion for (formal) verification closure
• Can be combined with simulation based verification and metrics
May 1, 2013
Thank You!
May 1, 2013
What about redundancy?
Redundant code cannot be observed!
• Redundant code will be marked uncovered if not treated specially.
What are sources of redundancy?
• Redundant logic not driving any outputs
• Safety logic meant to implement fail safe devices
How to mitigate?
• Identify redundant code and mark it
• Remove redundant code before running coverage
• Selectively activate/de-activate redundant parts
May 1, 2013
Functional vs. Structural Coverage Type Functional Structural
Control How much of the specified behavior has been activated?
How much of the DUV implementation has been activated?
Observation How much of the specified behavior has been verified?
How much of the DUV implementation has been verified?
Functional vs. Structural
• Functional - relates to specification
– Measures completeness of requirements verification
– Can identify gaps in verification plan
• Structural - relates to DUV
– Measures completeness of design intent verification
– Can identify unverified parts of DUV
Control vs. Observation
• Control - relates to activation
– Measures quality of stimuli
– Can identify unreachable code / over-constraining
• Observation - relates to causality
– Measures quality of checkers
– Can identify verification gaps
• Control coverage doesn’t imply any observation coverage.
• Observation coverage doesn’t imply any functional coverage.
• 100% observation coverage implies 100% control coverage.
• 100% Functional coverage + all assertions proven implies 100% observation coverage (if DUV contains no dead code, no constrained code, no redundant code)
May 1, 2013
Verification Environment with Exhaustive Coverage Analysis
DUV
Tests / Scenarios Checkers
Spec
Verification Environment
Verification
Plan
Coverage
Analysis
Control
Coverage
Observation
Coverage
Functional
Coverage
Have I written
enough stimuli?
Which parts of
my DUV have
been exercised?
Which parts of my
DUV have been
checked?
Did I write
enough checks?
Are all specified
functions
implemented?
Are all specified
functions verified?
Functional
Structural
May 1, 2013
What does block level coverage mean in the larger context?
System-level design
Block-level design (RTL)
Architecture-level design
Design start Verification closure
System-level simulation
Block-level simulation
Sub-system-level simulation
Formal ABV with Covage Analysis
UCDB
Aggregation of module / block level coverage results with higher level context
High-Level/RTL Design & Verification Flow