Lessons Learnt from the Adoption of Formal Model-based Development · 2012-05-08 · Lessons Learnt from the Adoption of Formal Model-based Development Alessio Ferrari1 Alessandro

Lessons Learnt from the Adoptionof Formal Model-based Development

Alessio Ferrari1 Alessandro Fantechi1,2

Stefania Gnesi1

1ISTI-CNR, Pisa, Italy2DSI, University of Florence, Florence, Italy

April 3, 2012

A. Ferrari, et al. (ISTI-CNR/DSI-UNIFI) Adoption of FM-based Development 1 / 20

OverviewDomain

Railway signallingStandard-regulated framework

TechnologiesFormal model-based development & Code generation

ToolsSimulink/Stateflow toolsuite

GoalsIdentification of a safe-subset of the modelling languageEvidence of the behavioural conformance between the generatedcode and the modelled specificationIntegration of the modelling and code generation technologieswithin the process that is recommended by the regulations

Tuning of the approach across 3 Projects


Formal model-based designFormal methods

Ten Commandments of Formal Methods...ten years later(Bowen, J.P., Hinchey, M.G., IEEE Computer, 1995-2006)

Formal methods are still perceived as experimental technologiesModel-based design

Model-based design is born later, but gained ground much fasterGraphical simulation is more intuitive than formal verificationSimulink/Stateflow is a de-facto standard

ChallengesHow to ensure that the generated code is compliant with themodelled application?How to integrate model-based practices with traditional certifiedprocesses?Formal model-based design


Problem Statement

HistoryMedium-size company operating in safety-related railwaysignalling systems developmentIntroduction of Simulink/Stateflow as design tools4 years research activity in collaboration with academia to adoptcode generation

Problem StatementDefine and implement a methodology for the adoptionof the code generation technology from formal modelsby a railway signalling company


Goals

Goal 1 - Modelling language restrictionSafety-critical code shall conform to specific quality standardsCompanies use coding guidelines in order to avoid usage ofimproper constructsIdentification of a safe subset of the modelling language

Goal 2 - Generated code correctnessProven-in-use translator requiredCompliance between generated code and model behaviour

Goal 3 - Process integrationIntroduction of new technologies in an established processrequires adjustments to the process structureProcess according to normative prescriptions


Projects


Automatic Train Protection (ATP) Systems


Project 1 History

PrototypingA Stateflow model was designed in collaboration with thecustomer in order to define and assess the system requirementsAssessment of the potentials of modelling for prototype definitionand requirements agreementSuccesful deployment of the system

Code Generation ExperimentsExperiments started when the system was already operationalDefinition of a first set of modelling guidelinesProper code synthesis of the single model units was achievedthrough Stateflow CoderNo V&V process still definedProject 1 remained an hand-crafted system


Project 2 History

Towards a V&V processTen times larger than Project 1 in terms of featuresInternal guidelines integrated with MAAB (Matlab AutomotiveAdvisory Board) recommendationsStatic anlaysis at model level for guidelines verificationFunctional unit-level verification

I Model-based testingI Abstract interpretation (Polyspace)

Evolution

Strict timing of the projectModel-based testing and guidelines verification only partiallyemployedAd-hoc solution for code verification


Project 3 History

A formal development processATP for metro signalling systemProject 1 < Project 3 < Project 2Hierarchical derivation approach with UML supportReal-time Workshop Embedded Coder adoptedFunctional unit-level verification

I Translation validation (back-to-back testing)I Abstract interpretation (Polyspace)

Experiments with formal verificationSimulink Design VerifierExperiments at module level95% of the requirements can be verified with the tool to achieve acost reduction of 50% to 60% in terms of man-hours


Goals


Projects vs Goals

Year Project Technologies ( Full or Partial Adoption) Goal

2007-2008 Project 1 Modelling guidelines (25) 1

Code generation (Stateflow Coder R2007b)

2008-2010 Project 2

Modelling guidelines + MAAB (43) 1

Code generation (Stateflow Coder R2007b) 2

Guidelines verification 3Model-based testing

Abstract interpretation (Polyspace 7.0)

2009-2011 Project 3

Modelling guidelines + MAAB (43) 1

Semantics restrictions 2UML + hierarchical derivation 3Code generation (RTW Embedded Coder R2010a)

Translation ValidationAbstract interpretation (Polyspace 8.0)

Formal Verification (Simulink Design Verifier R2010a)


Goal 1 - Modelling Language Restriction

Project 1 - Identify a proper subset of the languageI Analysis of the violations of the quality standard issued by the code

generated from the original modelI Definition of sub-models and evaluation of the translation of single

graphical constructsI Definition of a preliminary set of guidelines

Project 2 - A more general set of guidelinesI Previous guidelines could lack of generality since they were derived

from a specific modelI A comparison with the experience of other safety-critical domains

was needed −→ MAABI Guidelines oriented also to define well-structured models

Project 3 - Enable formal analysis and verificationI Reduction of the Simulink/Stateflow language to a semantically

unambiguous setI Inspired by the translation of Simulink/Stateflow into LustreI The models produced are independent from the simulation engine


Goal 2 - Generated Code Correctness

Project 2 - Preliminary definition of a verification approachProject 3 - Operational implementation of the approach

I Translation validationF Model/code back-to-back execution of unit testsF Comparison of the structural coverage obtained at model and at code

levelI Abstract interpretation

F Verification of runtime errorsF The tools implementing abstract interpretation work on a conservative

and sound approximation of the variable values in terms of intervalsF Finding errors in this larger approximation domain does not imply that

the bug also holds in the programF Problem of false positive casesF First step: large over-approximation set, in order to discover

systematic runtime errors and identify classes of possible falsepositives

F Second step: constrained abstract domain, derived from the firstanalysis, and the number of uncertain failure states to be manuallyreviewed is drastically reduced


Goal 3 - Process Integration

SW Requirements Phase

SW ArchitecturePhase

SW Module Design Phase

SW Validation Phase

SW IntegrationPhase

SW Module Test Phase

Code Phase




Model ArchitecturePhase

Model Module Design Phase

Model Module Test Phase

Model IntegrationPhase

Model Validation Phase

SW Validation Phase

SW IntegrationPhase


Code Phase







Model Integration

Phase


SW Validation Phase

SW IntegrationPhase


Code Phase

Model System Test Definition

Requirements Definition

UML Architecture

Stateflow Design

Model Unit Test Definition

Simulink Architecture

Unit Requirements Definition

SW System Test Definition







Model Integration

Phase


SW Validation Phase

SW IntegrationPhase


Code Phase

Model Unit Testing

Model Unit Testing Validation

SW Unit Test Generation

Model Integration

Model System Testing

Model System Testing Validation



UML Architecture

Stateflow Design


Model Static Analysis










Model IntegrationPhase


SW Validation Phase

SW IntegrationPhase


Code Phase

Code Generation (RTW Embedded

Coder)

Model Unit Testing



Model Integration





UML Architecture

Stateflow Design












Model Integration

Phase


SW Validation Phase

SW IntegrationPhase


Code Phase

SW Unit Testing

Code Generation (RTW Embedded

Coder)

Model Unit Testing



Model Integration



SW System Testing(HIL)

SW System Testing Validation

SW Integration



UML Architecture

Stateflow Design





SW Abstract Interpretation (Polyspace)



What We Learnt


What We Learnt

AbstractionI Models can be manipulated better than codeI Definition of test scenarios at module level without disrupting model

structureExpressiveness

I Graphical models are closer to the natural language requirementsI Unambiguous mean to exchange or pass artefacts among

developers

Cohesion & DecouplingI Interfaces among functionalities are based solely on dataI Control-flow is simplified since there is no cross-call among

different modules

UniformityI Generated code has a repetitive structure, which facilitates the

automation of the verification activitiesI One could look at the generated code as if it would be the software

always written by the same programmer


What We Learnt

TraceabilityI Software modules are directly traceable with the corresponding

blocks of the modelled specificationI Navigable links between the single code statements and the

requirements

ControlI Greater control over the componentsI Software with less bugs already before the verification activities

Verification CostI When passing from traditional code unit testing based on structural

coverage objectives, to testing based on functional objectives aidedwith abstract interpretation, it was possible to reduce the verificationcost of about 70%

I Recent experiments with formal verification have shown that thiscost can be further reduced by 50-66%


What We Learnt

PerformanceI The complexity of the generated code is higherI Optimization are not allowed for safety-critical softwareI More powerful hardware platforms are required

Manual Test DefinitionI Bottleneck of the verification processI 60-70% of the overall unit-level verification costI Formal verification can address this issue

Knowledge Transfer ProcessI Research assistant focused on the new technologyI Development team putting into practice the technologyI Balance between independence and participation


Conclusion

Introducing the formal design and code generation technologieswithin the development process of a railway signallingmanufacturerRadical changes in the verification processFormal model-based design has opened the door to model-basedtesting, has facilitated the adoption of abstract interpretation, andhas allowed performing the first successful experiences withformal verificationFour years and three projects to be defined and consolidatedIncremental tuning of the process also thanks to the flexibility ofthe toolsuiteResearch management model has been crucialWhat if UML-centered tools are going to be used?


Documents

Lessons Learnt from the Adoption of Formal Model-based Development · 2012-05-08 · Lessons Learnt from the Adoption of Formal Model-based Development Alessio Ferrari1 Alessandro