Verification of avionics

software to meet DO-178B/C

What, why and how?

Software V&V. NMI. Bristol. 10th November 2015

Dr. Guillem Bernat

bernat@rapitasystems.com

On-target Software Verification

Validation and Verification Validation: Checking that you are building the right thing

Verification: Checking that you are building the thing right

Software verification Process to determine that the software satisfies requirements

Does the software do what is supposed to do? On time?, all statements tested,

does not exceed stack limits, etc.

On-target: Information collected as close as possible to the final product.

Same object code as final system

Same target as final system

No models or assumptions needed

“Use evidence of the behaviour of the code running on target”

What is Worst-Case Execution Time

(WCET)?

WCET is longest possible execution time for a piece of

code on a specific hardware platform

An exact WCET is generally not feasible

Depends on SW architecture

SW structure

E.g. Data driven vs. structure driven programs

Hardware architecture

And other HW interactions

….

3

DO-178C 6.3.4: Reviews and Analyses of Source Code

These review and analysis activities … confirm that the Source Code satisfies these objectives:

…

f. Accuracy and consistency: The objective is to determine the correctness and consistency of the Source Code, including … worst-case execution timing, … The compiler (including its options), the linker (including its options), and some hardware features may have an impact on the worst-case execution timing and this impact should be assessed

Further clarified in DO-248B. FAQ#73

Note especially: Timing measurements by themselves cannot be used without an analysis demonstrating that the worst-case timing would be achieved, but processor behavior should be assessed.

Why measure WCET for aerospace applications?

4

Objective Activity Applicability by

Software Level

Description Ref Ref A B C D

Source Code complies with low-level

requirements

6.3.4.f 6.3.4

Manual analysis and measurement:

Review code to identify worst-case path

Devise test cases

Instrument code

Measure execution time

Consider the use of a "safety margin"

Identifying the worst-case path is difficult:

Source code might be misleading: Simple assignment statements may take a long time to execute

Complex-looking groups of statements might take little (or no) time to execute

Loop bounds can be difficult to determine

Mutually exclusive code paths

Measuring is difficult:

• Excluding interference from other tasks/partitions/cores

• Repeating timing measurements as problems are observed and investigated

What is current practice?

5

WCET analysis strategies

Definitions Root: the piece of code we are doing timing analysis for. E.g. Task, interrupt, etc.

Arguments of what is WCET High Watermark: The longest observed execution “end-to-end” of Root (e.g. task)

Requires finding out the test case that drives the worst case path and worst-

case hardware usage.

WCET *:Approximation to WCET by only partial instrumentation up to “n” levels

deep in the call tree.

WCET (deep): Analytical calculation of the longest execution time of Root. Based

on measurements of small sections of code and analytical path calculations.

Timing bands

Band I

Band II

Band III

Band IV

Challenges to WCET collection… and ways to address them

Two main sources of challenges to collecting worst-case

execution time

System constraints

"Performance improvement" features

System constraints:

Limited target resources/limited data bandwidth…

… addressed by flexible instrumentation

"Performance improvement" features include:

"locally deterministic" hardware acceleration features (pipelines, branch prediction)

… collect multiple instances of timing data

cache memory (data, instruction)

… follow guidance of CAST-20

Multicore

… measure on a core-by-core basis, against no load and a high load

RTOS

… instrument context switches, demultiplex traces

8

Using hybrid analysis to improve measurements

WCET analysis process

9

Obtain HWM (High Water

Mark) – longest observed

execution time

Integrating into the build process

10

Integrating into the build process

11

Collecting Verification Data

12

Tool Qualification Issues

According to DO-330, tool qualification is required if: • The tool could fail to detect an existing error

• The tool's output could not be verified by another activity

• Processes are eliminated, reduced or automated by the tool

The "applicant" needs to provide: • PSAC

• TOR

• TAS

• TVR/TOVVR

• TQP

DO-330 process can be used to provide Qualification Kits for

both DO178-B and DO178-C… but….

13

Structural Code Coverage

Measurement Notes

Function

coverage

Each function has been called at least once

Not required by DO-178B/C

Call coverage Each function has been called at least once, and each

different function call has been encountered at least

once

Not required by DO-178B/C

Statement

coverage

Each statement in the code has been encountered at

least once

Required for DO-178B/C Level A,

B, C

Decision

coverage

Each decision in the code has evaluated true at least

once and evaluated false at least once, and each

function entry and exit point has been encountered at

least once

Required for DO-178B/C level A,

B

Condition

coverage

Each condition in the code has evaluated true at least

once and evaluated false at least once

Not required by DO-178B/C

Modified

Condition/

Decision

Coverage

Decision coverage plus each condition has been shown

to independently affect the outcome of its enclosing

decision

Required for DO-178B/C level A

MC/DC

Requirements based

testing

“flip” a condition and it

should affect the outcome

of the decision

Very hard to test from

requirements

Hard to define the “missing”

test vectors

Decisions

In “if”, “while”, …

In assignments

A= B && C;

In function calls

F(B&&C);

…

Case study 1

The problem On-target code coverage

Avionics display system

500 KLOC of Ada code

Tests on rig + host

Previous method required 15 builds due to overheads of instrumentation

6 days to run full suite x 15 = a lot of time

The solution Highly optimised target integration using RapiCover

Only one build, overall overhead below 15%.

Merging evidence for coverage from host build and on-target build

The benefit Reduce time and cost to perform structural code coverage

Code coverage performed at every single test build

(as opposed to once at the end)

Case study 2

The problem WCET analysis of engine management system

Large Ada code base

Customer own processor (new version of the CPU)

Internal static analysis tool could not be adapted to new processor

The solution Using built-in logic analyser interface to provide very low overhead tracing

mechanism

Develop a hardware tracing solution

Using RapiTime for detailed timing of blocks of code as part of “unit tests”.

Using RapiTime to compute WCET analytically using timings from unit tests

The benefit WCET analysis produced as a “side effect” of unit testing

Tool Qualification for DO178B.

Case study 3

The problem Code coverage on-target

Ethernet switch on aircraft. Mixed C/Ada code base.

Code coverage during “power cycles”.

Manual effort to review “coverage holes”

The solution Lightweight integration of RapiCover

Using LiveMaps recorded using RTBx

Using Justification workflow for addressing coverage holes

The benefits Fully automated structural code coverage analysis from on-target tests

Justifications used once, reused automatically for each build

Elimination of manual analysis of power-cycle code

Elimination of manual code reviews

Summary

Verification activities to satisfy DO178-B/C

Timing

Multiple definitions of what WCET is!.

Analytical WCET possible only when supported by design decisions

Structural code coverage

Very well understood

Future challenges, more demanding evidence from certification authorities.

More emphasis by certification authorities for evidence from tests “on-target”