Validation and Verification - its-mobility.de · Introduction − Using models and (semi-)formal...

Validation and

verification of

specification models

Test4Rail, Braunschweig

Dr. Oliver Lemke

V2.0

Agenda

− Introduction

− Needs

− Process

− Conclusion

18.10.2017 2

18.10.2017 3

SIGNON business activities

Planning Technical Consulting Engineering

Signalling systems

Telecommunications

Power supply

Systems

Software

Safety

Studies

Methodology

Processes

Introduction

− Using models and (semi-)formal languages like SysML have become more common

over the last years for specifying railway systems:

▪ NeuPro – DB’s standardisation of interlocking architecture in Germany

▪ EULYNX – European counterpart

− EULYNX (https://www.eulynx.eu) is an initiative of 12 European infrastructure

managers (IMs) to harmonize interlocking architectures

− This is accomplished by creating unified operator’s specification documents for the

supply industry

18.10.2017 4

NO_OPERATING_VOLTAGE

BOOTING FALLBACK_MODE

OPERATIONAL

INITIALISING

OPERATING_VOLTAGE_SUPPLIED

F_EST_SubS_TDS - Behaviour [SubSTDS STD1]

when( T1_P ower_On_detec ted )/when( T2_P ower_Off_detec ted )/

when( T5_S IL_not_fulfilled )/

when( T3_Res et )/

when( T5_SIL_not_fulfilled )/

T12_Interrupt_Safe_Communication_Protocol_Connection := true;

when( T3_Res et )/

when( T4_Booted )[D13_S tart_Tim e_s ync hronisation_is_initiated]/

when( T9_Proces s_Data_Interface_connec tion_established )/

when( T5_SIL_not_fulfilled )/

T12_Interrupt_Safe_Communication_Protocol_Connection := true;

when( T3_Res et )/

when( T7_Invalid_or_m iss ing_Configurat ion_Data_carrier )/

when( T10_Safe_Communication_Protocol_Connection_disconnected

)/

SysML state machine diagram Simulation interface

Specification document

18.10.2017 5

Agenda

− Introduction

− Needs

− Process

− Conclusion

18.10.2017 6

Needs

− The claim is, that model-based, (semi-)formal specifications improve the

specification quality by:

▪ Being correct

▪ Being consistent

▪ Being unambiguous

− But this is only true if, the underlying model is unambiguous, consistent and correct

− As models can be reused for system acceptance tests, integration tests and various

simulations, these requirements for model quality are even aggravated.

Hence assuring a high level of quality for the models becomes essential.

18.10.2017 7

Agenda

− Introduction

− Needs

− Process

− Conclusion

18.10.2017 8

Process - simplified CENELEC V-model

Implementation

phase

Validation

Verification

P1

P5

18.10.2017 9

P9

Process - small V-model for specification phase

State machine (STM)

implementation

STM

acceptance

Validation

Verification

User

requirements

Formalised

requirements

18.10.2017 10

User

Reqs. Formalised reqs.

State machine

implementation

State machine

acceptance

System Env.

Stimulus A

Response a

Response d

Stimulus B

Stimulus C

Scenarios as SysML

sequence diagrams (ca. 20 –

50 scenarios per subsystem)

NO_OPERATING_VOLTAGE

BOOTING FALLBACK_MODE

OPERATIONAL

INITIALISING

OPERATING_VOLTAGE_SUPPLIED

F_EST_SubS_TDS - Behaviour [SubSTDS STD1]

when( T1_P ower_On_detec ted )/when( T2_P ower_Off_detec ted )/

when( T5_S IL_not_fulfilled )/

when( T3_Res et )/

when( T5_SIL_not_fulfilled )/

T12_Interrupt_Safe_Communication_Protocol_Connection := true;

when( T3_Res et )/

when( T4_Booted )[D13_S tart_Tim e_s ync hronisation_is_initiated]/

when( T9_Proces s_Data_Interface_connec tion_established )/

when( T5_SIL_not_fulfilled )/

T12_Interrupt_Safe_Communication_Protocol_Connection := true;

when( T3_Res et )/

when( T7_Invalid_or_m iss ing_Configurat ion_Data_carrier )/

when( T10_Safe_Communication_Protocol_Connection_disconnected

)/

SysML state machines in

modelling tool Executable simulator

Know-

ledge

Informal

documents

Creation - Modeller

Verification - Tester

Validation - Stakeholder

18.10.2017 11

Process - verification

Verification step 1 (Black-Box-Verification):

− Verify that state machines (STM) react as specified in sequence diagrams (SD)

− Implemented by stimulating the executable simulator according to the SDs and

reading back its responses, comparing them to the responses defined in the SDs

− Test execution is highly automated through GUI testing tools (e.g. Ranorex)

Verification step 2 (White-Box-Verification):

- Verify that STM does not add implicit behaviour not specified in sequence diagrams

- Checked by verifying that all SDs fully cover the STM according to defined coverage

criteria, e.g. full state and transition coverage

- Unmarked states and transitions are not covered by sequences and therefore describe

additional behaviour

- Generate SDs covering the missing elements in STM and discuss with stakeholder

18.10.2017 12

NO_OPERATING_VOLTAGE

BOOTING FALLBACK_MODE

OPERATIONAL

INITIALISING

OPERATING_VOLTAGE_SUPPLIED

F_EST_SubS_TDS - Behaviour [SubSTDS STD1]

when( T1_P ower_On_detec ted )/when( T2_P ower_Off_detec ted )/

when( T5_S IL_not_fulfilled )/

when( T3_Res et )/

when( T5_SIL_not_fulfilled )/

T12_Interrupt_Safe_Communication_Protocol_Connection := true;

when( T3_Res et )/

when( T4_Booted )[D13_S tart_Tim e_s ync hronisation_is_initiated]/

when( T9_Proces s_Data_Interface_connec tion_established )/

when( T5_SIL_not_fulfilled )/

T12_Interrupt_Safe_Communication_Protocol_Connection := true;

when( T3_Res et )/

when( T7_Invalid_or_m iss ing_Configurat ion_Data_carrier )/

when( T10_Safe_Communication_Protocol_Connection_disconnected

)/

18.10.2017 13

Process - validation

Know-

ledge

Informal

documents

Operational stakeholders validate the STM against the input documents by using their own

test cases. This assures diversity and coverage of domain knowledge.

18.10.2017 14

Agenda

− Introduction

− Needs

− Process

− Conclusion

18.10.2017 15

Experiences

Statistics for a SysML model of the interface between interlocking and axel counting

system.

Model

− Number of SDs as formalised requirements: 45

− Number of states in the STMs: 15

− Number of transitions in the STMs: 38

Detected errors (after multiple manual reviews and quality checks)

− Verification - functional errors (STM does not match SDs): 5

− Verification - implicit behaviour (STM contains behaviour not specified in SDs): 3

− Validation - functional errors: 3

18.10.2017 16

Conclusion

− The process presented is able to improve the specification quality

− The quality of model-based specifications is typically higher than of text-based

specifications

− The additional benefit of using formal verification techniques must be evaluated, as

the efforts for applying them in the real world are still very high

Thank you!

18.10.2017 17