V | 1999-12-31

VectorCAST (Code Test) Product Line Simon Watson, FAE

Software Test Automation for High Quality Software

2

Some cars from Ryton on Dunsmore, Coventry are 100% free of software defects by design.

Why do we test with VectorCAST?

3

VC/QA adds value to existing system/functional tests egacross the network interface of the ECU

>The customer has (hopefully!) invested in system tests but how do they measure the completeness of the test suite?

>System tests mostly execute the “happy path”.

>Is our customer (the end user) the first person to execute some of the code – how can we be sure?

>In practice you don’t get 100% code coverage from a suite of system tests even when you have 100% requirements coverage.

Where can we test with VectorCAST/QA – 1a?

4

The starting point for VectorCAST/QA is an existing application (in C/C++/Ada) that compiles and links and runs.

We capture structural code coverage data from the system test execution.> We do source level coverage because “the bug is in the source code”.

> We instrument the source code so that we capture the execution of functions, statements and decision points, sub-conditions and MCDC patterns, function calls.

>There is a limited amount of data to store so a high-throughput debug port is not necessary.

We could also collect other data (eg function call parameters and timing/stack information) but that is rarely done.

Where can we test with VectorCAST/QA – 1b?

5

Where can we test – architecture of demo 1

6

The demo shows VC/QA driving VI/CANoe to execute system tests inside a DLL simulation of the actual target>No debugger is necessary for VectorCAST and the system test works the same way as it does without VectorCAST.

>We can use VectorCAST to drive CANoe (via COM) or CANoeto drive VectorCAST (via command line interface).

In the example, the code to test is generated from a model, to control the interior lighting. But it can be any C, C++ or Ada source file(s).

Where can we test with VectorCAST/QA – demo 1

7

Demo 1 : CANoe tests and interleaved data transfer

8

Demo 1 : Completion step in vTestStudio

9

Demo 1 : coverage report

10

The starting point for VC/C/C++/Ada is a compilable source or header file (much less onerous than for QA).

A unit/module/whitebox test is at the API point(s) of an individual function>A unit test tool allows us to execute a function (or a chain or

sequence of functions) in isolation by stubbing external interactions.

>Probably the only way to test boundary values and fully exercise all decision points (system tests usually execute the happy path)

>Often hard to make the correspondence between requirements and software lines of code but this exposes the weakness of the requirements, not the tests.

>We can write and run test cases before we write the code.

Where can we (unit/module) test with VectorCAST -2a?

11

A unit/module test usually stubs the hardware and RTOS

>Our light control code is normally stimulated/exercised by CAN messages but we don’t need the CAN to drive the “finite state machine”.

>So we can develop and run most unit tests first on the host system.

>But endianness and the sizes of types and the FPU (and the compiler) may be different so we will also test on the target and your safety standard might also require on-target module testing.

>And we can run the unit test as a task under an RTOS (coffee time question – is there advantage in doing that?).

Where can we test -2b?

12

A unit/module test should not “look inside” a function>That would make it hard to maintain when the code changes.

>A function should be designed so that it isn’t necessary to access the local variables

>But it is possible and sometimes it is necessary because of particularly “difficult” code constructs> const objects

> inline access to hardware

> function scope statics

A unit test makes it easy to “inject errors” to test paths that are otherwise “almost unreachable”.> if ( today == blue_moon && today == preston_guild ) {}

Where can we test -2c?

15

Unit/Module Testing, who provides what?

Unit Testing Overview

Unit(s)

Under

Test

Stubbed

Dependents

Real

DependentsSource

Code

Units

(.c, .cpp)

Test

Driver

16

The demo shows a VC/C unit test environment for the same source file as we used in the system test demo 1

>We will import the coverage from the system tests and then look at some test cases that complete the coverage

Where can we test with VectorCAST – demo 2

17

Demo 2 : unit test case input and expected values

18

VectorCAST Unit Test framework and the target connection

19

The exhibition stand demo shows the same VC/C unit test environment as demo 2 (same code and test cases) but this time we will use an embedded compiler and target board via a USB-JTAG.

VectorCast supports the board/cpu via the compiler and debugger so it is the toolchain that matters more than the silicon.

Where can we test : demo 3

23

“Everyone” does (should do) static analysis but…

…where is the compiler?> Static analysis does not check the compiler output

…where are the requirements? code can be 100% Misra compliant and 100% wrong at the same time.

if ( reverse_requested != 0 )

{

if ( ( speed < 10 ) || ( speed > -10 ) )

{

engage_reverse();

}

}

Why isn’t static analysis enough?

24

Do I need the CAN interface connected to show that my slew rate filter implementation is correct?

>Yes, my customer only gave me CAN-level test data

>Surely (truly, hopefully…) there is a better (cheaper, quicker, more thorough) answer???

>The slew rate filter should be independent of the IO implementation

Coffee time question

25

This is architecturally equivalent to the first demo but using a board rather than a simulation. The demo stand showed an Arm Cortex-M4 board with CAN bus>but the important factor is a network connection to avoid the use of a debugger which may not be present anyway in a functional/system test.

>We will “system test” the LEDs and collect the coverage over the CAN bus and show the coverage report.

Demo 4

26

Final demo screenshot 1

27

Final demo screenshot 2