Upload
adacore
View
1.797
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
Challenges of Agile QualificationTowards a Qualifying Machine
Challenges of Agile QualificationTowards a Qualifying Machine
Matteo Bordin
Matteo Bordin
How does it feel to qualify a tool for DO-178?How does it feel to qualify a tool for DO-178?
The Big Freeze problemThe Big Freeze problem
Is Tool Qualification equivalent to Baselining?
Verify a slightly different coding standard
Change the code generation strategy
Regenerate a source-to-obj traceability study
The Big Freeze problem (II)The Big Freeze problem (II)
A new bug is found!
Please, do NOT fix it!
Really cheaper/safer to invent workarounds?
The Big Freeze problem (III)The Big Freeze problem (III)
So, what is our goal?
AdaCore engineers improve our tech every day
Constraining engineers’ creativity cannot work
…still, we target the high-integrity domain
QUALIFICATION
How to Avoid the Big Freeze Problem (I)How to Avoid the Big Freeze Problem (I)
Continuous Qualification
Mirrors the “Continuous Integration” concept
Qualification artifacts evolve along source code
A tool shall always be in a “semi-qualifiable” state
Maximize automation of qualification-oriented activities
How to Avoid the Big Freeze Problem (II)How to Avoid the Big Freeze Problem (II)
Delta Qualification
What is the minimum effort to re-achieve qualifiability?
Which artifact shall be (re-)produced
Which activities shall be (re-)performed
An example of Continuous CertificationAn example of Continuous Certification
Requirement R1
Test Case TC1
Source File SF1
Test Case TC2
Test Case TC3
Test Procedure TP1
Test Procedure TP2
Test Procedure TP3
Structural Coverage Report SCR1
Coding Standard CS1
Coding Standard Compliance Report
CSC1
Modified
Re-generated
Re-generated
A modification of an artifact
triggers the qualification
infrastructure to execute a set
of activities
(possibly overnightly)
Continuous Certification + Delta CertificationContinuous Certification + Delta Certification
Sounds nice eh?
Requires an integrated certification infrastructure
The topic of this talk!
Focus on automation & optimization
Difficult to generalize for different teams/clients
A drastically different way of working
The core problemThe core problem
Establishing a dependency & traceability graph
To use to trigger automated activities
To calculate the Delta Qualification effort
Low creation & maintenance effort
Understand when an artifact has changed
Propagate un-valid status
Presenting current results…
…as well as un-implemented ideas
A good vision of our way of thinking…
…and how we develop TQL5 kits
From now on, this is a long-term view
Introducing the Qualifying MachineIntroducing the Qualifying Machine
An intelligent repository for qualification artifacts
Integrated with build/test infrastructure
Provides hooks to invoke scripts to trigger activities
Support for change impact analysis
QM back-end
Qualifying Machine Architecture (I)Qualifying Machine Architecture (I)
SVN, Git, … Repository
Traceability tracking
Dependency tracking
Action tracking
Un-validity propagation
Automated activity triggering
QM front-end
Development Engineer
Process Engineer
Project Manager
Certification Authority
Testing/Building infrastructure
Qualifying Machine Architecture (II)Qualifying Machine Architecture (II)
Major design decisions & challenges
What to track (artifacts, activities, …)
How to build a traceability & dependency graph
How to understand if an artifact has changed
Un-validity propagation algorithm
What do we track?What do we track?
Everything is an artifact!
Each artifact is a (set of) file(s)
Metadata are placed in a single folder
Activities ALWAYS produce qualification artifacts
An activity not producing an artifact IS NOT WORTH IT
State of an activity = state of the artifacts it produces
Leverage as much as possible on naming conventions
Configuring the QMConfiguring the QM
project <NAME> is
artifact Requirement is
files: “req.txt”
activity Decompose produces
derived : Requirement [0..*] in “./Req_<name>”
activity Write_Test_Case produces
test_cases : Test_Case [1..*] in “./TC_<name>”
activity QA_Review produces
qa_report : Report [0..1] in “qa.txt”
traceability:
standard : Doc [1] in “<ROOT>/Standards/req_std.txt”
Building a traceability & dependency graphBuilding a traceability & dependency graph
My_Project
Req_SysReq1
Req_SwReq1
TC_Flying
TC_Departure
QA
Req_SysReq1
Req_SwReq1
Configuration File
project <NAME> is artifact Requirement is files: “req.txt”
activity Decompose produces derived : Requirement [0..*] in “./Req_<name>”
activity Write_Test_Case produces test_cases : Test_Case [1..*] in “./TC_<name>”
+ =
When does an artifact change?When does an artifact change?
At the heart of un-validity propagation
Timestamp, SVN diff, MD5, …
How to cope with complex artifacts (UML or Simulink models)?
How to identify changes not requiring formal re-validation?
Example: a typo
TC1
6.3.2bcd
Un-validity propagation algorithm (II)Un-validity propagation algorithm (II)
LLR1
6.4.4.1
Re-validate 6.4.4.1
(test coverage of LLR),
possibly editing TC1
Re-measure
structural coverage
TP1
Source File SF1
Manual activity
Depending on the status of artifacts & activities, the QM:- Proposes a minimal list of manual activities- Triggers automated activities- Track each single action of the user
LLRs are accurate, consistent,
compatible and verifiable
TORs
Un-validity propagation algorithm (I)Un-validity propagation algorithm (I)
Known Constraints and Open Problems
References, but does not depends on
Tool Qualification Plan
Depends on
Verification Results Analysis
Delivery File
Modifying the Known Constraints and
Open Problems triggers the production
of a minimal set of manual activities to be
performed in order to re-achieve
qualification:
- on TORS
- on elements derived from TORs
How do we use the QM? (I)How do we use the QM? (I)
Not targeting the typical DOORS users…
Lightweight artifact editing/managing
How do we use the QM? (II)How do we use the QM? (II)
Un-validity propagation feedback
How do we use the QM? (III)How do we use the QM? (III)
How do we fill the QM repository? (IV)How do we fill the QM repository? (IV)
However, we understand other tools are used
Develop a set of plug-in
Import artifact metadata in the QM
The QM focuses on continuous/delta qualification
Integrate external dev. tools in the QM
Wrap-upWrap-up
Our experience
From a semi-agile infrastructure to the QM
Qualification kit for a verification tool (TQL5): can we scale?
Lessons learnt
Fragmentation & Automation!
What about workflow modeling/tracking?
Future developments
www.forge.open-do.org/projects/qmachine
SafeCer Project (Artemis)
Presented byPresented by