certkitiec_sltest_certreport

8/20/2019 certkitiec_sltest_certreport

1/13

Distribution, copying or any other use of information in this report in part is strictly prohibited.

Report

on the

Certificate

Z10 15 06 67052 016

Software Tools for Safety Related Development

Simulink ®

Test™

Manufacturer

The MathWorks, Inc.3 Apple Hill Drive

Natick, MA, 01760-2098

USA

Report No. MN86842CRevision 1.0 dated 2015-06-15

Testing Body

TÜV SÜD Rail GmbHEmbedded Systems

Certification Body

TÜV SÜD Product Service GmbHRidlerstraße 6580339 Munich


2/13

TÜV SÜD Rail GmbH Report No.: MN86842CEmbedded Systems Revision 1.0Barthstr. 16 S. Waldhausen80339 München 2015-06-15

Phone: +49 89 5791-4378; Fax: -2933 Page 2 of 13

Revision Log

Rev. Date Name Changes/History

1.0 2015-06-15 S. Waldhausen, M. Braun Initial Report for Release R2015b


3/13


Phone: +49 89 5791-4378; Fax: -2933 Page 3 of 13

Content Page

1

PURPOSE AND SCOPE ...................................................................................................... 4

2

PRODUCT OVERVIEW ....................................................................................................... 4

2.1 General Description ....................................................................................................... 4

2.2 Scope ............................................................................................................................ 5

3 IDENTIFICATION ................................................................................................................. 6

4 CERTIFICATION .................................................................................................................. 6

4.1 Standards ...................................................................................................................... 6

4.2 Basis of certification ...................................................................................................... 6

5 RESULTS ............................................................................................................................ 7

5.1 Software development and quality engineering processes ............................................ 7

5.2

Customer bug reporting processes ................................................................................ 7

5.3 Requirements on software tools in IEC 61508, ISO 26262, and EN 50128.................... 8 5.3.1 General .................................................................................................................. 8

5.3.2 Simulink ® Test™ .................................................................................................... 9

5.4 Tool classification and validation according to IEC 61508 ........................................... 10

5.5 EN 50128 .................................................................................................................... 11

5.6 Tool classification and qualification according to ISO 26262 ....................................... 11 5.6.1 Estimation of TD and resulting TCL: ..................................................................... 11

5.6.2 Evaluation of the tool development process ......................................................... 12

5.6.3 Validation of the software tool............................................................................... 12

5.6.4

Summary .............................................................................................................. 12

5.7 IEC 62304 ................................................................................................................... 13

6 GENERAL CONDITIONS AND RESTRICTIONS ............................................................... 13

7 SUMMARY AND CERTIFICATE NUMBER ....................................................................... 13


4/13


Phone: +49 89 5791-4378; Fax: -2933 Page 4 of 13

1 Purpose and scope

TÜV SÜD Rail GmbH evaluated the Simulink® Test™ product of The MathWorks, Inc. The sectionsof the MathWorksTM development organization responsible for the Simulink® Test™ product have

been audited to assess their development and quality assurance procedures.Recurring evaluations focus on processes used by the Simulink® Test™ teams to implement en-hancements and modifications, as well as quality engineering, and customer bug reporting process-es.

The aim of the assessment was to determine the suitability for use in development processes whichneed to comply with IEC 61508, ISO 26262 or EN 50128. The assessment also covered tool classi-fication and tool qualification measures according to ISO 26262.

The basic assessment is documented in the Technical Report MN86843T, recent modifications willbe reported in Modification Reports according to the table below.

Title Document Name Date RevisionTechnical Report on Functional Safety MN86843T -V1.0.pdf 12.06.2015 1.0

2 Product overview

Simulink® Test™ is a verification tool for authoring, managing, and executing systematic, simulation-based tests of the Simulink models.

2.1 General DescriptionSimulink® Test™ includes a test sequence block to construct test sequences and assessments, anda test manager to manage and execute tests. It enables functional, baseline, equivalence, and back-to-back testing, including software-in-the-loop (SIL) and processor-in-the-loop (PIL). The tool alsoallows generating reports, archiving and reviewing test results, rerunning failed tests, and debuggingthe component or system under test.


5/13


Phone: +49 89 5791-4378; Fax: -2933 Page 5 of 13

2.2 Scope

The testing for the certification of Simulink® Test™ focused on the following use-cases, as describedin the IEC Certification Kit files (see section 3):

[SLTEST_UC1] Development and execution of tests for Simulink models

[SLTEST_UC2] Development and execution of tests for back-to-back testing between modeland code

[SLTEST_UC3] Assessment of test results

[SLTEST_UC4] Generation of test reports

[SLTEST_UC5] Identification of traceability between requirements and tests cases

The assessment covered the following capabilities of the Simulink® Test™ tool, which support the

accomplishment of the above listed use cases: Development of test harness for subsystem or model testing

Specifying sequence of tests using Test Sequence block

Specifying pass-fail criteria, including tolerances, limits, and temporal conditions

Implementation of baseline, equivalence, and back-to-back testing

Development setup and cleanup scripts for customizing test execution

Authoring, executing, and organizing test cases and their results using Test manager

Automatic report generation to document test outcomes


6/13


Phone: +49 89 5791-4378; Fax: -2933 Page 6 of 13

3 Identification

Release DateSimulink

®

Test™

Reference Workflow Documentation:

IEC Certification Kit

R2015b Sept. 2015 1.1 Simulink® Test™ Reference Workflow, V3.6 Simulink® Test™ ISO 26262 Tool Qualification Package, V3.6

4 Certification

4.1 Standards

Standard Description

IEC 61508-1:2010 Functional Safety of electrical/electronic/programmable electronicsafety-related systems Part 3: General requirements

IEC 61508-3:2010 Functional Safety of electrical/electronic/programmable electronicsafety-related systems Part 3: Software requirements

ISO 26262-8:2011 Road vehicles – Functional safety – Part 8: Supporting processes‘Confidence in the use of software tools‘

EN 50128:2011 Railway applications – Communications, signalling and processingsystems – Software for railway control and protection systems

4.2 Basis of certification

Software development, quality engineering, and customer bug reporting processes

Requirements on software tools in IEC 61508, ISO 26262, or EN 50128

Tool classification and validation according to IEC 61508

Tool classification and qualification according to ISO 26262


7/13


Phone: +49 89 5791-4378; Fax: -2933 Page 7 of 13

5 Results

5.1 Software development and quality engineering processes

The software development and quality engineering processes applied for Simulink® Test™ havebeen audited, no objections were found.

To ensure adherence to the software development and quality engineering processes, as well as tokeep track of quality improvements, the processes to implement enhancements and modificationsare audited once a year by TÜV SÜD.

Product versions that are released in between two consecutive audits are subject to a defined ap-proval procedure by TÜV SÜD. The procedure includes the following elements:

The MathWorks, Inc. documents new customer visible features for each release in the corre-sponding release notes.

The MathWorks, Inc. documents enhancements and new features of each Simulink® Test™ version in an internal delta report.

Test procedures for enhancements and new features are referenced in the delta report todocument MathWorks internal validation activities for newly developed features.

5.2 Customer bug reporting processes

MathWorks reports known critical bugs brought to its attention on its bug report system athttp://www.mathworks.com/support/bugreports/. The bug reports are an integral part of the docu-mentation for each release.

The bug report system provides an interface for customers to view and submit bug reports. Custom-ers can track the status of open bugs. Customers can choose to receive notifications for new or up-dated bug reports. The bug reports on this web site include internally as well as externally nominat-ed bugs. If applicable, bug reports include provisions for known workarounds or file replacements.

Customers can use the bug report mechanism to nominate bugs. These nominations are processedand evaluated by The MathWorks, Inc. development organization.

http://www.mathworks.com/support/bugreports/http://www.mathworks.com/support/bugreports/


8/13


Phone: +49 89 5791-4378; Fax: -2933 Page 8 of 13

5.3 Requirements on software tools in IEC 61508, ISO 26262, and EN 50128

5.3.1 General

ISO 26262, IEC 61508 and EN 50128 in their current versions contain explicit requirements on soft-ware tools.

They strongly recommend the application of development tools and provide provisions for usingmodel-based design for software development. At the same time, they demand to perform an analy-sis of the tools used, and an analysis on how they are embedded in the development process:

analysis of tool usage (IEC 61508)

analysis of tool use cases (ISO 26262)

analysis on the effect of possible malfunctions of the applied tool(s).

Depending on the outcome of the above analysis, the standards referred to above demand

a) fault mitigation measures (process)

b) the qualification, respectively validation of tools.

These activities should complement each other, and the combination of both shall reduce the num-ber of faults impacting the final product to a minimum.


9/13


Phone: +49 89 5791-4378; Fax: -2933 Page 9 of 13

5.3.2 Simulink ® Test ™

The verification tool allows the automation of core verification and validation activities for Simulinkmodels and generated code. The following use cases reflect activities that are required in a software

development process according to the Functional Safety Standards ISO 26262, IEC 61508 and EN50128:






The use cases involve the capabilities of Simulink® Test™ listed in section 2.2:

Development and execution of tests for Simulink models and for back-to-back testing betweenmodel and code:- Create and execute test harnesses for subsystem or model testing- Specifying sequence of tests using Test Sequence block- Specifying pass-fail criteria, including tolerances, limits, and temporal conditions- Implementation of baseline, equivalence, and back-to-back testing- Development setup and cleanup scripts for customizing test execution

Assessment and generation of test reports containing simulation and test results:- Authoring, executing, and organizing test cases and their results using Test manager- Automatic report generation to document test outcomes

- Identification of traceability between requirements and tests case

The aim of the testing was to certify the involved tool capabilities for the use in development pro-cesses which need to comply with IEC 61508, ISO 26262 or EN 50128.


10/13


Phone: +49 89 5791-4378; Fax: -2933 Page 10 of 13

5.4 Tool classification and validation according to IEC 61508

Simulink® Test™ is a class T2 off-line support tool.

The following list provides considerations on how tool users are being supported w.r.t. the require-ments of IEC 61508-3 clause 7.4.4:

Simulink® Test™ can be integrated with other Model-Based Design and verification toolsfrom The MathWorks, Inc. (cf. IEC 61508-3, 7.4.4.2, Note 3). A representative combination oftools is tested at the manufacturer’s site. (cf. IEC 61508-3, 7.4.4.9, 7.4.4.18 a).

The tool documentation for Simulink® Test™ (cf. IEC 6158-3, 7.4.4.4) is provided with theproduct.

Each release of the tool is identifiable (cf. IEC 61508-3, 7.4.4.15 a).

MathWorks reports critical known bugs brought to its attention on its bug report system at

http://www.mathworks.com/support/bugreports/ (cf. IEC 61508-3, 7.4.4.6, Note 1). The Release Notes provide the version history for Simulink® Test™. Tool users can assess

available bug reports for different tool versions via the bug reports system (cf. IEC 61508-3,7.4.4.6, Note 1)

The MathWorks, Inc., as well as 3rd party vendors, offer training courses for MathWorks tools(cf. IEC 61508-3, 7.4.4.2, Note 6).

The MathWorks, Inc. developed and applied validation suites to validate the model compli-ance checking and model coverage analysis capabilities. The application of these validationsuites helps to uncover potential bugs in Simulink® Test™.

Test procedures for enhancements/new features are referenced in the delta report to docu-ment MathWorks internal validation activities for newly developed features. The MathWorks,Inc. validated Simulink® Test™ and provided documentation of this validation to TÜV SÜD forreview and approval (cf. IEC 61508-3, 7.4.4.6, 7.4.4.7).

Summary:

All Simulink® Test™ versions listed in section 3 are certified as T2 off-line support tools and are suit-able for safety-related use in application development up to SIL 3 according to IEC 61508:2010. Thetools meet the requirements of IEC 61508-3 7.4.4 to the extent applicable to a tool manufacturer.The certification covers the following capabilities:

Develop test harness and test procedure

Generate test reports containing simulation and test results, including requirement traceability

The tool classification and the assessment of the tool validation activities were carried out by TÜVSÜD.

Tool certification can be claimed by referencing this certification report and the corresponding certifi-cate.

http://www.mathworks.com/support/bugreports/http://www.mathworks.com/support/bugreports/


11/13


Phone: +49 89 5791-4378; Fax: -2933 Page 11 of 13

5.5 EN 50128

EN 50128:2011 is an application standard derived from IEC 61508. The requirements for softwaretools are explicitly derived from the requirements on software tools according to IEC 61508-3:2010.

Due to the equivalences between the two standards no separate testing has been performed withrespect to EN 50128.

Simulink® Test™ are suitable to be used in the development of safety-related software according toEN 50128:2011 up to SIL 3/4. Tool certification for the versions listed in section 3 can be claimed byreferencing this certification report and the corresponding certificate.

5.6 Tool classification and qualification according to ISO 26262

The tool classification according to ISO 26262 depends on the particular use-cases used during thedevelopment of safety-related application software components.

For Simulink® Test™, the following use-cases were considered in the tool classification process:






Based on these use cases, the tool impact of Simulink® Test™ is TI2.

5.6.1 Estimation of TD and resu lt ing TCL:Develop test harness and test procedure

[SLTEST_UC1], [SLTEST_UC2]:

Provided that the error prevention or detection measures listed in the reference workflow forSimulink® Test™ are carried out, the capability of Simulink® Test™ to develop test harness and testprocedure has been classified as TCL1.

Generate test reports containing simulation and test results, including requirement traceability[SLTEST_UC3], [SLTEST_UC4], [SLTEST_UC5]:

Assuming that there are no systematic measures in the development process to verify the generatedtest reports, the tool error detection for the capability of Simulink® Test™ to generate test reports

containing simulation and test results, including requirement traceability is TD3. The resulting toolconfidence level is TCL3.

A combination of the following tool qualification methods was carried out for the capability ofSimulink® Test™ to generate test reports containing simulation and test results, including require-ment traceability:

Evaluation of the tool development process Validation of the software tool


12/13


Phone: +49 89 5791-4378; Fax: -2933 Page 12 of 13

5.6.2 Evaluat ion of the tool development process

TÜV SÜD conducts yearly surveillance audits of the software development and quality engi-neering processes for Simulink® Test™.

The MathWorks, Inc. documents new customer visible features for each release in the corre-sponding release notes. The release notes were submitted to TÜV SÜD.

The MathWorks, Inc. documents enhancements and new features for each release to bequalified in a comprehensive delta report. The delta reports were submitted to TÜV SÜD.

5.6.3 Validation of the so ftware too l

The MathWorks, Inc. developed and applied a validation suite for the capability of Simulink® Test™ to generate test reports containing simulation and test results, including requirementtraceability that can be used to validate these features. The application of this validation suitehelps to uncover potential bugs in Simulink® Test™. A successful validation is considered as

a means of end-to-end validation of the capability of Simulink® Test™ to generate test re-ports containing simulation and test results, including requirement traceability. The validationreports were submitted to TÜV SÜD.

Test procedures for enhancements/new features of Simulink® Test™ are referenced in thedelta report to document The MathWorks, Inc. internal validation activities for newly devel-oped features.

5.6.4 Summary

All Simulink® Test™ versions listed in section 3 are qualified for all ASILs according to ISO 26262.

The qualification comprises the following capabilities: Develop test harness and test procedure

Generate test reports containing simulation and test results, including requirement traceability

The capability of Simulink® Test™ to generate test reports containing simulation and test results,including requirement traceability has been classified as TCL3 and qualified accordingly.

Provided that the error prevention or detection measures listed in the reference workflow forSimulink® Test™ are carried out, the capability of Simulink® Test™ to develop test harness and testprocedure has been classified as TCL1. The tool qualification measures have been carried out on a

voluntary basis to provide additional confidence.The review of the tool classifications and the assessment of the results of the measures applied toqualify the software tool were carried out by TÜV SÜD.

Tool qualification for Simulink® Test™ can be claimed by referencing this certification report and thecorresponding certificate.


13/13


5.7 IEC 62304

IEC 62304:2006 provides a framework of life cycle processes for the safe design and maintenanceof medical device software.

IEC 62304 does not place specific requirements on software tools, or on the qualification of tools,but IEC 62304 advises that “IEC 61508 can be looked to as a source of methods, tools and tech-niques that can be used to implement the requirements in IEC 62304” (IEC 62304:2006, C.1).

6 General conditions and restrictions

As a prerequisite to claim tool qualification for Simulink® Test™ according to ISO 26262, theerror prevention or detection measures listed in the respective reference workflows shall beapplied.

7 Summary and certificate number

This report specifies the conditions of use and restrictions required for the application of Simulink® Test™ by The MathWorks, Inc. on the certificate:

Z10 15 06 67052 016

Munich, 2015-06-15

Technical CertifierPeter Weiß

Documents

certkitiec_sltest_certreport