See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/261228227

Is the current DO-254 verification process adequate for the future?

Conference Paper · October 2012

DOI: 10.1109/DASC.2012.6383061

CITATION

1

READS

69

1 author:

Some of the authors of this publication are also working on these related projects:

Maritime RobotX View project

monocopter View project

Brian Butka

Embry-Riddle Aeronautical University

21 PUBLICATIONS   45 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Brian Butka on 25 August 2015.

The user has requested enhancement of the downloaded file.

Is the Current DO-254Verification ProcessAdequate forthe Future?

Brian ButkaEmbry-Riddle Aeronautical University

•Requirements Based Verification• DO-254 is based on Requirements Based Verification (RBV).

• Review requirements for correctness and completeness.

• Write verification cases for every requirement.• Run the verification suite in simulation and analyze results.

• RBV is adequate for DAL C designs. Additional analysis/verification is required for DAL A/B designs

•Requirements Based Verification• Data from the semiconductor industry

• Directed testing is adequate for small to medium complexity designs.

• Does not scale well.• As the design complexity grows

• Humans are no longer able to anticipate all possible interactions

• Computer tools are needed to supplement the directed tests

•Simple Design Requirements1. When the BTNU is pushed and SW0 is

high, LED7 is lit. LED7 is dark for all other combinations of the BTNU andSW0.

2. LED 0 is lit if BTNL or BTNR is pressed. 3. LED4 is lit when SW1 and SW2 are

different. If they are the same, LED 3 must be lit.

•Simple Design1. When the BTNU is pushed and SW0 is high,

LED7 is lit. LED7 is dark for all other combinations of the BTNU and SW0.

LED7<=BTNU^SW0; (Exclusive or)1. LED 0 is lit if BTNL or BTNR is pressed.

LED<=1 ; LED0 is always on2. LED4 is lit when SW1 and SW2 are different. If

they are the same, LED 3 must be lit.LED4<=1;LED3<=1; LED4 and LED3 always on

•Multi-Year Survey• I have presented these requirements to 120 students in my FPGA design class.

• So far every student interpreted requirement #2 as requiring an OR gate and requirement #3 as requiring XOR and XNOR gates.

• Human assumptions and biases in interpreting the requirements are difficult to prevent and detect.

•Constrained Random Verification• A method of supplementing RBV is constrained random verification. (CRV)

• The hardware requirements are entered into an automated tool.

• If a condition is not prohibited, then it is allowed. • Random test cases are generated.• Tests are searching a large space so want to generate as many test cases as possible

• Constrained Random Verification should be applied early and often.

•Constrained Random Verification• When constrained random testing is applied early in the design process it is very effective at identifying problems with the requirements such as :

• Incomplete Specifications• Vagueness• Omissions

• Random tests are not subject to human biases.

• CRV Tries unusual combinations: • Repeated resets• Odd sequences of inputs

•Robustness Testing• Constrained Random Verification is often expanded to include conditions outside of what is allowed by the requirements.

• For example, in a packet based system:• Incorrect packet payload sizes• CRC errors• No end of packet delimiter• Missing packets

•Elemental Analysis• For DAL A/B designs additional analysis is required to determine if all of the design elements have been verified

• Elements are one or more lines of code.• Elemental analysis looks to make sure that every line of code has executed.

• Statement Coverage

•Statement coveragealways @(posedge clk)

out<=resetb&(A||B);

This means out = resetb and (A or B)

• A single test case say resetb=0, A=0, and B=0; will generate 100% statement coverage.

• All of the elements are tested?

•Improved Coverage Metrics• Improved coverage metrics have been introduced such as Modified Condition Decision Coverage (MCDC) which thoroughly test all combinations of inputs and output decisions.

•MCDC• MCDC generates an exponentially increasing number of test cases.

• Works well in single process environments.• When there are multiple interacting processes running in parallel, MCDC verification is of limited value.

• Processes can overwrite data in other processes.• If the timing is wrong it does not matter if the logic is correct.

Coverage Metrics based on Hardware Elements • Toggle coverage

• After the testbench has run, every signal in the design has transitioned from 0 to 1 and 1 to 0.

• Finite State Machine Coverage• After the testbench has run, every reachable state has been reached and every state transition has been executed at least once.

•Challenges For All Metrics• In safety critical designs, there is often hardware that only operates if a hardware failure (such as a single event upset) has occurred.

• Requires specially modified simulations to model the expected hardware failure.

• Is the modified simulation comprehensive and correct?

•Functional Verification• Functional Verification is a extension of Requirements Based Verification that addresses the following functional coverage points:

• Were all possible input stimuli variations injected?

• Were all possible output conditions achieved? • Did all possible internal system state transitions take place?

• Did all the interesting events occur?

•Interesting Events• Interesting events are conditions or states that require extra scrutiny.

• Interesting events can be identified by either the verification team or the design team.

• Typical interesting events of focus on:• Concurrent events• Timing variations• Clock domain crossings• Anywhere the designers focused their efforts.

•A weakness of Requirements Based and Functional Testing• An error is only detected if the error propagates to an output.

• We would like to detect errors even when they do not result in an erroneous output.

• A different combination of inputs and system state could allow the error to propagate to an output and cause an error in the final system.

• Assertions can be used to detect errors at the source.

•Assertions• A designer may utilize and 8 bit register to contain a count between 0 and 10.

• The designer would write the assertionassert (Count<11)to document their intent.

• If count ever exceeds 10 the hardware is operating in regime that the designer did not anticipate/allow.

• It is flagged as an error regardless of whether an erroneous output occurs.

•Assertion Coverage• If a designer believes a condition is significant enough to document with an assertion, then the test bench should cover all of the assertions.

•Multiple Layers of Coverage Metrics• The testbench can be evaluated on a number of coverage metrics

• Requirements Based Verification• Functional Verification• Assertion Based Verification

• Each of these metrics should achieve 100% coverage.

•Achieving Complete Coverage• Although it is possible to achieve coverage with carefully written directed tests it is preferable to achieve coverage using constrained random testing.

• Repeated application of constrained random testing including robustness tests will evaluate each of the requirements functional coverage points and assertions multiple times under differing conditions.

•A Recommended Verification Strategy (Cadence Design Systems)

•Coverage Domains• Three plan has three parallel coverage domains

• Code Coverage• Assertion Coverage• Functional Coverage

• Tests written by the designer are shown as “Smoke Tests” and are not counted towards coverage goals.

•Timeline• The timeline labels include:

• Testbench Ready• Feature Finished• Bug Rate Leveled• IP Integration Complete

• Different verification techniques are applied at different times.

•Hardware Verification is Last• If the RTL code is being edited, it does not make sense to run hardware metrics since the hardware implementation is not complete.

• Once the RTL is finalized, the hardware coverage is evaluated.

• Directed high level system tests are run in parallel with the hardware verification.

•A Survey of Verification Methods

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Other

OVM/UVM

VMM

System C

System Verilog

Functional Coverage

Code Coverage

Formal Methods

Assertions

Utilization

Utilization

•Conclusion• The combination of Requirements Based Testing and Elemental Analysis is a core part of hardware verification.

• Assertions and Hardware Verification Metrics should be used to enhance the verification process

View publication statsView publication stats