Upload
marian-gilmore
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
EE694v-Verification-Lect5 -1-
Lecture 5 - Verification Tools
• Automation improves the efficiency and reliability of the verification process
• Some tools, such as a simulator, are essential. Others automate tedious tasks and increase confidence in the outcome.
• It is not necessary to use all the tools.
EE694v-Verification-Lect5 -2-
Lecture Overview
• Will look at the other tools used in verification– Third party models
– Hardware modelers
– Waveform viewers
– Code coverage tools
– Verification languages
– Revision control
– Configuration management
– Issue tracking
– Metrics
EE694v-Verification-Lect5 -3-
Third Party Models
• Many designs use off the shelf parts• To verify such a design, must obtain a model to
these parts• Often must get the model from a 3rd party• Most 3rd party models are provided as compiled
binary models• Why buy 3rd party models?
– Engineering resources
– Quality (especially in the area of system timing)
EE694v-Verification-Lect5 -4-
Hardware Modelers
• Are for modeling new hardware. Some hardware may be too new for models to available– Example: In 2000 still could not get a model of the Pentium
III
• Sometimes cannot simulate enough of a model in an acceptable period of time
EE694v-Verification-Lect5 -5-
Hardware Modelers (cont)
• Hardware modeler features– Small box that connects to network that contains a real copy
of the physical chip
– Rest of HDL model provides inputs to the chip and obtains the chips output to return to your model
EE694v-Verification-Lect5 -6-
Waveform Viewers
• Lets you view transitions on multiple signals over time
• The most common of verification tools• Waveform can be saved in a trace file• In verification
– need to know expected output and whenever the simulated output
is not as expected • both the signal value and the signal timing
– use the testbench to compare the model output with the expected
EE694v-Verification-Lect5 -7-
Code Coverage
• A technique that has been used in software engineering for years.
• By covering all statements adequately the chances of a false positive (a bad design tests good) are reduced.
• Never 100% certain that design under verification is indeed correct. Code coverage increases confidence.
• Some tools may use file I/O aspect of language and others have special features built into the simulator to report coverage statistics.
EE694v-Verification-Lect5 -8-
Adding Code Coverage
• If built into simulator - code is automatically instrumented.
• If not built in - must add code to testbench to do the checking
EE694v-Verification-Lect5 -9-
Code Coverage
• Objective is to determine if you have overlooked exercising some code in the model– If you answer yes then must also ask why the code is present
• Coverage metrics can be generated after running a testbench
• Metrics measure coverage of– statements
– possible paths through code
– expressions
EE694v-Verification-Lect5 -10-
Statements and Blocks
• Statement coverage can also be called block coverage
• The Model Sim simulator can show how many times a statement was executed
• Also need to insure that executed statements are simulated with different values
• And there is code that was not meant to be simulated (code specifically for synthesis for example)
EE694v-Verification-Lect5 -11-
Path Coverage
• Measures all possible ways you can execute a sequence of statements
• Example has four possible paths
EE694v-Verification-Lect5 -12-
Path Coverage Goal
• Desire is to take all possible paths through code• It is possible to have 100% statement coverage but
less than 100% path coverage• Number of possible paths can be very, very large
=> keep number of paths as small as possible• Obtaining 100% path coverage for a model of
even moderate complexity is very difficult
EE694v-Verification-Lect5 -13-
Expression Coverage
• A measure of the various ways paths through code are taken
• Example has 100% statement coverage but only 50% expression coverage
EE694v-Verification-Lect5 -14-
100% Code Coverage
• What do 100% path and 100% expression coverage mean?– Not much!! Just indicates how thoroughly verification suite
exercises code. Does not indicate the quality of the verification suite.
– Does not provide an indication about correctness of code
• Results from coverage can help identify corner cases not exercised
• Is an additional indicator for completeness of job– Code coverage value can indicate if job is not complete
EE694v-Verification-Lect5 -15-
Verification Languages
• Verilog and VHDL were designed as design languages• Verification languages are designed for verification
– e/Specman from Verisity– VERA from Synopsys– RAVE from Cronology
• Even with a verification language still– need to plan verification– design verification strategy and design verification architecture– create stimulus– determine expected response– compare actual response versus expected response
EE694v-Verification-Lect5 -16-
Revision Control
• Need to insure that model verified is model used for implementation
• Managing a HDL-based hardware project is similar to managing a software project
• Require a source control management system• Such systems keep last version of a file and a
history of previous versions along with what changes are present in each version
EE694v-Verification-Lect5 -17-
Configuration Management
• Wish to tag (identify) certain versions of a file so multiple users can keep working
• Different users have different views of project
EE694v-Verification-Lect5 -18-
File Tags
• Each file tag has a specific meaning
EE694v-Verification-Lect5 -19-
Issue Tracking
• It is normal and expected to find functional irregularities in complex systems
• Worry if you don’t!!! Bugs will be found!!!
• An issue is anything that can affect the functionality of the design– Bugs during execution of the testbench
– Ambiguities or incompleteness of specifications
– A new and relevant testcase
– Errors found at any stage
• Must track all issues if a bad design could be manufactured were the issue not tracked
EE694v-Verification-Lect5 -20-
Issue Tracking Systems
• The Grapevine– Casual conversation between members of a design team in which
issues are discussed
– No-one has clear responsibility for solution
– System does not maintain a history
• The Post-it System– The yellow stickies are used to post issues
– Ownership of issues is tenuous at best
– No ability to prioritize issues
– System does not maintain a history
EE694v-Verification-Lect5 -21-
Issue Tracking Systems (cont.)
• The Procedural System– Issues are formally reported
– Outstanding issues are reviewed and resolved during team meetings
– This system consumes a lot of meeting time
• Computerized Systems– Issues seen through to resolution
– Can send periodic reminders until resolved
– History of action(s) to resolve is archived
– Problem is that these systems can require a significant effort to use
EE694v-Verification-Lect5 -22-
Code Related Metrics
• Code Coverage Metrics - how thoroughly does verification suite exercise code
• Number of Lines of Code Needed for Verification Suite - a measure of the level of effort needed
• Ratio of Lines of Verification Code to Lines of Code in the Model - measure of design complexity
• Number of source code changes over time
EE694v-Verification-Lect5 -23-
Quality Related Metrics
• Quality is subjective• Examples of quality metrics
– Number of known outstanding issues
– Number of bugs found during service life
• Must be very careful to interpret and use any metric correctly!!!