Program Validation and Testing

7/31/2019 Program Validation and Testing

1/21


2/21

An embedded system that has a marginal timing problemor a cross-talk problem can appear to work correctly forlong stretches of time and then just die.

When the right combination of 1s and 0s appears on theright bus at the right time, a glitch occurs, and a bit flipswhere it shouldnt, taking the system down with it.

To avoid these kinds of problems we go forProgram Validation

Program testing


3/21

Validation loosely refers to the process ofdetermining that a design is correct.

Simulation remains the main tool to validate amodel, but the importance of post-productionverification is growing, especially for safety ofcritical embedded systems.

Although still in its infancy, it shows more promisethan verification of software programs, becauseembedded systems are often specified in a more

restricted way.


4/21

Simulating embedded systems is challenging

because they are heterogeneous.

Most ES contain both software and hardwarecomponents that must be simulated at the sametime, which is the co-simulation problem.

execute the software as fast as possible,

often on a host machine that may be faster thanthe final embedded CPU

Keep the hardware and software simulations

synchronized, so that they interact just as they willin the target system.


5/21

General-purpose software simulator (VHDL or

Verilog) to simulate a model of the target CPU,executing the software program on this simulationmodel.

Different models can be employed,

Gate-level models

Instruction-set architecture

Bus-functional models

Translation-based models


6/21

Its an approach which keeps track of timein software and hardware independentlyand using various mechanisms itsynchronizes them periodically.


7/21

Their are two basic mechanisms forsynchronizing time in hardware and software.

Software is the master and hardware is the slave.In this case, software decides when to send amessage, tagged with the current software clock

cycle, to the hardware simulator. Depending onthe relation between software and hardware time,the hardware simulator can either continuesimulation until software time or back-up the

simulation to software time

Hardware is the master and software is the slave.In this case, the hardware simulator directly calls

communication procedures which, in turn, callsoftware code.


8/21

Formal verification is the process of

mathematically checking that the behavior of asystem, described using a formal model, satisfiesa given property, also described using a formal

model.

Specification Verification

Implementation Verification

The properties we check are traditionally brokeninto two classes

Safety properties

Liveness properties


9/21

Embedded systems software testing shares much

in common with application software testing.

But some important differences exist betweenapplication testing and embedded systems testing.

Embedded developers often have access tohardware-based test tools that are generally notused in application development

Also, embedded systems often have uniquecharacteristics that should be reflected in the testplan.

These differences tend to give embedded systems

testing its own distinctive flavor.


10/21

Before we begin to design tests, its important to

have a clear understanding of why we are testing.This understanding influences which tests westress and how early we begin testing.

In general, we test for four reasons:

To find bugs in software (testing is the only way to

do this)

To reduce risk to both users and the company

To reduce development and maintenance costs

To improve performance


11/21

One of the important results from theoretical

computer science is a proof (known as theHalting Theorem) that its impossible to prove thatan arbitrary program is correct. Given the right

test, however, you can prove that a program isincorrect (that is, it has a bug). Its important toremember that testing isnt about proving thecorrectness of a program but about finding

bugs. Experienced programmers understand that every

program has bugs.

The only way to know how many bugs are left ina program is to test it with a carefully designed


12/21

Testing minimizes risk to yourself, your company,and your customers.

The objectives in testing are to demonstrate toyourself that the system and software workscorrectly as designed.

You want to be assured that the product is assafe as it can be.

I.e., you want to discover every conceivable faultor weakness in the system and software beforeits deployed in the field.


13/21

The classic argument for testing comes from QualityManagement.

In 1990, HP sampled the cost of errors in softwaredevelopment during the year. The answer, $400

million, which shocked HP into a completely neweffort to eliminate mistakes in writing software.

If half of it is spent in the labs on rework and half of itin the field to fix the mistakes that escaped from thelabs, which will amounted to one-third of thecompanys total R&D budget and could haveincreased earnings by almost 67%.


14/21

The earlier a bug is found, the less expensive itis to fix. The cost of finding errors and bugs in areleased product is significantly higher thanduring unit testing.


15/21

Testing maximizes the performance of thesystem. Finding and eliminating dead code

and inefficient code can help ensure that thesoftware uses the full potential of thehardware and thus avoids the dreadedhardware re-spin.


16/21

We want to test every possible behavior in our

program. This implies testing every possiblecombination of inputs or every possible decisionpath at least once.

The basic approach is to select the tests thathave the highest probability of exposing an error.

Functional Tests

Coverage Tests

Gray-Box Testing


17/21

Functional testing is often called black-box testing

because the test cases for functional tests aredevised without reference to the actual code.

Black-box tests are based on what is known about

which inputs should be acceptable and how theyshould relate to the outputs.

Black-box tests know nothing about how thealgorithm in between is implemented.

Because black-box tests depend only on theprogram requirements and its I/O behavior, theycan be developed as soon as the requirements are

complete.


18/21

Stress tests

Boundary value tests

Exception testsError guessing

Random tests

Performance tests


19/21

Coverage tests attempt to avoid this weakness in

functional tests by ensuring that each codestatement, decision point, or decision path isexercised at least once.

Also known as white-box tests.

Statement coverage

Decision or branch coverage

Condition coverage


20/21

white-box tests can be intimately connected to

the internals of the code, they can be moreexpensive to maintain than black-box tests.

black-box tests remain valid as long as the

requirements and the I/O relationships remainstable,

Tests that only know a little about the internalsare called gray-box tests. Gray-box tests can bevery effective when coupled with error guessing.


21/21

Embedded software must run reliably without

crashing for long periods of time. Embeddedsoftware is often used in applications in whichhuman lives are at stake.

Embedded systems are often so cost-sensitivethat the software has little or no margin forinefficiencies of any kind.

Embedded software must often compensate forproblems with the embedded hardware.

Real-world events are usually asynchronous andnondeterministic, making simulation tests difficult

and unreliable Your company can be sued if yourcode fails

Documents

Program Validation and Testing