Challenges of Agile Qualification

Challenges of Agile QualificationTowards a Qualifying Machine

Challenges of Agile QualificationTowards a Qualifying Machine

Matteo Bordin

bordin@adacore.com

Matteo Bordin

bordin@adacore.com

How does it feel to qualify a tool for DO-178?How does it feel to qualify a tool for DO-178?

The Big Freeze problemThe Big Freeze problem

Is Tool Qualification equivalent to Baselining?

Verify a slightly different coding standard

Change the code generation strategy

Regenerate a source-to-obj traceability study

The Big Freeze problem (II)The Big Freeze problem (II)

A new bug is found!

Please, do NOT fix it!

Really cheaper/safer to invent workarounds?

The Big Freeze problem (III)The Big Freeze problem (III)

So, what is our goal?

AdaCore engineers improve our tech every day

Constraining engineers’ creativity cannot work

…still, we target the high-integrity domain

QUALIFICATION

How to Avoid the Big Freeze Problem (I)How to Avoid the Big Freeze Problem (I)

Continuous Qualification

Mirrors the “Continuous Integration” concept

Qualification artifacts evolve along source code

A tool shall always be in a “semi-qualifiable” state

Maximize automation of qualification-oriented activities

How to Avoid the Big Freeze Problem (II)How to Avoid the Big Freeze Problem (II)

Delta Qualification

What is the minimum effort to re-achieve qualifiability?

Which artifact shall be (re-)produced

Which activities shall be (re-)performed

An example of Continuous CertificationAn example of Continuous Certification

Requirement R1

Test Case TC1

Source File SF1

Test Case TC2

Test Case TC3

Test Procedure TP1

Test Procedure TP2

Test Procedure TP3

Structural Coverage Report SCR1

Coding Standard CS1

Coding Standard Compliance Report

CSC1

Modified

Re-generated

Re-generated

A modification of an artifact

triggers the qualification

infrastructure to execute a set

of activities

(possibly overnightly)

Continuous Certification + Delta CertificationContinuous Certification + Delta Certification

Sounds nice eh?

Requires an integrated certification infrastructure

The topic of this talk!

Focus on automation & optimization

Difficult to generalize for different teams/clients

A drastically different way of working

The core problemThe core problem

Establishing a dependency & traceability graph

To use to trigger automated activities

To calculate the Delta Qualification effort

Low creation & maintenance effort

Understand when an artifact has changed

Propagate un-valid status

Presenting current results…

…as well as un-implemented ideas

A good vision of our way of thinking…

…and how we develop TQL5 kits

From now on, this is a long-term view

Introducing the Qualifying MachineIntroducing the Qualifying Machine

An intelligent repository for qualification artifacts

Integrated with build/test infrastructure

Provides hooks to invoke scripts to trigger activities

Support for change impact analysis

QM back-end

Qualifying Machine Architecture (I)Qualifying Machine Architecture (I)

SVN, Git, … Repository

Traceability tracking

Dependency tracking

Action tracking

Un-validity propagation

Automated activity triggering

QM front-end

Development Engineer

Process Engineer

Project Manager

Certification Authority

Testing/Building infrastructure

Qualifying Machine Architecture (II)Qualifying Machine Architecture (II)

Major design decisions & challenges

What to track (artifacts, activities, …)

How to build a traceability & dependency graph

How to understand if an artifact has changed

Un-validity propagation algorithm

What do we track?What do we track?

Everything is an artifact!

Each artifact is a (set of) file(s)

Metadata are placed in a single folder

Activities ALWAYS produce qualification artifacts

An activity not producing an artifact IS NOT WORTH IT

State of an activity = state of the artifacts it produces

Leverage as much as possible on naming conventions

Configuring the QMConfiguring the QM

project <NAME> is

artifact Requirement is

files: “req.txt”

activity Decompose produces

derived : Requirement [0..*] in “./Req_<name>”

activity Write_Test_Case produces

test_cases : Test_Case [1..*] in “./TC_<name>”

activity QA_Review produces

qa_report : Report [0..1] in “qa.txt”

traceability:

standard : Doc [1] in “<ROOT>/Standards/req_std.txt”

Building a traceability & dependency graphBuilding a traceability & dependency graph

My_Project

Req_SysReq1

Req_SwReq1

TC_Flying

TC_Departure

QA

Req_SysReq1

Req_SwReq1

Configuration File

project <NAME> is artifact Requirement is files: “req.txt”

activity Decompose produces derived : Requirement [0..*] in “./Req_<name>”

activity Write_Test_Case produces test_cases : Test_Case [1..*] in “./TC_<name>”

+ =

When does an artifact change?When does an artifact change?

At the heart of un-validity propagation

Timestamp, SVN diff, MD5, …

How to cope with complex artifacts (UML or Simulink models)?

How to identify changes not requiring formal re-validation?

Example: a typo

TC1

6.3.2bcd

Un-validity propagation algorithm (II)Un-validity propagation algorithm (II)

LLR1

6.4.4.1

Re-validate 6.4.4.1

(test coverage of LLR),

possibly editing TC1

Re-measure

structural coverage

TP1

Source File SF1

Manual activity

Depending on the status of artifacts & activities, the QM:- Proposes a minimal list of manual activities- Triggers automated activities- Track each single action of the user

LLRs are accurate, consistent,

compatible and verifiable

TORs

Un-validity propagation algorithm (I)Un-validity propagation algorithm (I)

Known Constraints and Open Problems

References, but does not depends on

Tool Qualification Plan

Depends on

Verification Results Analysis

Delivery File

Modifying the Known Constraints and

Open Problems triggers the production

of a minimal set of manual activities to be

performed in order to re-achieve

qualification:

- on TORS

- on elements derived from TORs

How do we use the QM? (I)How do we use the QM? (I)

Not targeting the typical DOORS users…

Lightweight artifact editing/managing

How do we use the QM? (II)How do we use the QM? (II)

Un-validity propagation feedback

How do we use the QM? (III)How do we use the QM? (III)

How do we fill the QM repository? (IV)How do we fill the QM repository? (IV)

However, we understand other tools are used

Develop a set of plug-in

Import artifact metadata in the QM

The QM focuses on continuous/delta qualification

Integrate external dev. tools in the QM

Wrap-upWrap-up

Our experience

From a semi-agile infrastructure to the QM

Qualification kit for a verification tool (TQL5): can we scale?

Lessons learnt

Fragmentation & Automation!

What about workflow modeling/tracking?

Future developments

www.forge.open-do.org/projects/qmachine

SafeCer Project (Artemis)

Presented byPresented by