Qualkitdo Rptgenext Tcp

8/20/2019 Qualkitdo Rptgenext Tcp

1/25

DO Qualification Kit

Simulink® Report Generator ™ Test Cases and Procedures

R2015b, September 2015


2/25

How to Contact MathWorks

Latest news: www.mathworks.com

Sales and services: www.mathworks.com/sales_and_services

User community: www.mathworks.com/matlabcentral

Technical support: www.mathworks.com/support/contact_us

Phone: 508-647-7000

The MathWorks, Inc.

3 Apple Hill Drive Natick, MA 01760-2098

DO Qualification Kit: Simulink ® Report Generator ™ Test Cases and Procedures

© COPYRIGHT 2010 – 2015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under

the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written

consent from The MathWorks, Inc.

FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the

federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees

that this software or documentation qualifies as commercial computer software or commercial computer software documentation

as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms andconditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,

reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or

other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.

If this License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the

government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.

Trademarks

MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks f or alist of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective

holders.

Patents

MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents f or moreinformation.

http://www.mathworks.com/http://www.mathworks.com/http://www.mathworks.com/sales_and_serviceshttp://www.mathworks.com/sales_and_serviceshttp://www.mathworks.com/matlabcentral/http://www.mathworks.com/matlabcentral/http://www.mathworks.com/support/contact_us/http://www.mathworks.com/support/contact_us/http://www.mathworks.com/trademarkshttp://www.mathworks.com/trademarkshttp://www.mathworks.com/trademarkshttp://www.mathworks.com/patentshttp://www.mathworks.com/patentshttp://www.mathworks.com/patentshttp://www.mathworks.com/patentshttp://www.mathworks.com/trademarkshttp://www.mathworks.com/support/contact_us/http://www.mathworks.com/matlabcentral/http://www.mathworks.com/sales_and_serviceshttp://www.mathworks.com/


3/25

Revision History

September 2010 New for Version 1.3 (Applies to Release 2010b)

April 2011 Revised for Version 1.4 (Applies to Release 2011a)September 2011 Revised for Version 1.5 (Applies to Release 2011b)

March 2012 Revised for Version 1.6 (Applies to Release 2012a)

September 2012 Revised for Version 2.0 (Applies to Release 2012b)


September 2013 Revised for Version 2.2 (Applies to Release 2013b)


October 2014 Revised for Version 2.4 (Applies to Release 2014b)


September 2015 Revised for DO Qualification Kit Version 3.0 (Applies to Release 2015b); renamed to Simulink ®

Report Generator™ Test Cases and Procedures


4/25


5/25

v

Contents

1

Introduction ...................................................................................................................................... 1-1

2

Test Cases and Procedures ............................................................................................................... 2-1

2.1

System Design Description Report ......................................................................................... 2-1

2.2

Model XML Comparison Report ............................................................................................ 2-1

3

Traceability Between Operational Requirements and Test Cases and Procedures ........................... 3-1

3.1

System Design Description Report – Traceability Matrix....................................................... 3-2

3.2

Model XML Comparison Report – Traceability Matrix ......................................................... 3-6

4

Executing Test Cases and Procedures .............................................................................................. 4-1

4.1

Executing Test Cases and Procedures for System Design Description Report ....................... 4-1

4.2

Executing Test Cases and Procedures for Model XML Comparison Report .......................... 4-3

5 Test Results ...................................................................................................................................... 5-1

5.1 System Design Description Report ......................................................................................... 5-2 5.2

Model XML Comparison Report ............................................................................................ 5-4


6/25

vi


7/25

1 Introduction

This document comprises test cases and procedures (Reference DO-330 Sections 10.3.3 and

10.3.4) for the following capability of the Simulink ® Report Generator ™ verification tool:

System Design Description report

Model XML Comparison report

This document is intended for use in the DO-330 tool qualification process for TQL-5 tools.

See also the DO Qualification Kit User's Guide, R2015b.


8/25

1-2


9/25

2 Test Cases and Procedures

Requirements- based test cases and procedures have been developed from the “System Design

Description Report Operational Requirements” and “Model XML Comparison Report

Operational Requirements” sections in the following document:

Simulink Report Generator Tool Operational Requirements

To access the tool operational requirements document, on the MATLAB® command line, type

qualkitdo to open the Artifacts Explorer. The document is in Simulink Report Generator.

2.1 System Design Description ReportThe test cases and procedures exercise the System Design Description (SDD) report capability,

and reside in the Simulink ® test models identified in System Design Description Report – Traceability Matrix. The test procedure applies each SDD report capability to a subset of the

preceding test models, and resides in the report generator file identified in System Design

Description Report – Traceability Matrix.

2.2 Model XML Comparison ReportThe test cases and procedures exercising the Model XML Comparison report capability include

Simulink ® Report Generator setup file qualkitdo_xmlcomp_rs.rpt and associated m-

files, mat-files, and Simulink test models. Test files and their allocation to the tool operationalrequirements are identified in Model XML Comparison Report – Traceability Matrix.

The report setup file invokes MATLAB code that sequentially modifies values of each

qualifying parameter of the test model using pre-generated test values and runs XMLcomparison between the original and modified model. Generated comparison reports areevaluated to confirm that the reports contain the correct information for the corresponding

parameters. Qualification status is assigned to PASS if all generated reports are correct. The

summary qualification test report summarizes the outcome of the tests run and include lists of


10/25

2-2

the qualified parameters in the appendixes. Figure 1 shows the Model XML Comparison testing

workflow.

Figure 1: Test Workflow


11/25

3 Traceability Between OperationalRequirements and Test Casesand Procedures


12/25

3-2

3.1 System Design Description Report – TraceabilityMatrixThe test cases and procedures cover the Tool Operational Requirements for the System Design

Description (SDD) report, as shown in the following table. The test models are used by the test

procedure qualkitdo_rptgenext_tcp.rpt .

System Design Description Report – Traceability Matrix

Test Model Requirement ID ExpectedResult

slvnvdemo_fuelsys_docreq.slxslvnvdemo_fuelsys_htmreq.slx

SDD_mdlverSDD_checksum

SDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authorsSDD_blkdiag

SDD_execorderSDD_name

SDD_path

SDD_SF_chartSDD_dataSDD_SFtarget

SDD_reqsSDD_configset

SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOC

SDD_rptformat_RTF

Included in pre-generated

reports

sldemo_mdlref_basic.slx SDD_mdlver

SDD_checksumSDD_userdef_titleSDD_userdef_subtitle

SDD_userdef_authorsSDD_blkdiagSDD_blkparasSDD_execorder

SDD_mdlrefSDD_configset

SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOCSDD_rptformat_RTF

Included in

pre-generatedreports


13/25

3-3


SDD_cont.slx

SDD_discont.slxSDD_discrete.slx

SDD_logic.slx

SDD_lookup.slx

SDD_math.slx

SDD_other.slx

SDD_signals.slx

SDD_sources.slx


SDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors

SDD_blkdiagSDD_blkparas

SDD_mdlverSDD_checksumSDD_userdef_title

SDD_userdef_subtitleSDD_userdef_authorsSDD_blkdiagSDD_data

SDD_nameSDD_path

SDD_chksumSDD_SF_chartSDD_SFttSDD_eml

SDD_SFtargetSDD_configset

SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOC

SDD_rptformat_RTF


reports


14/25

3-4


SDD_eml_call_gf.slx SDD_mdlver

SDD_checksumSDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors

SDD_blkdiagSDD_outputs

SDD_execorderSDD_nameSDD_path

SDD_SF_chartSDD_SFgraphfuncSDD_SFdataSDD_eml

SDD_SFtargetSDD_configset


Included in


SDD_SF_ArithLogic.slx

SDD_SF_data_check.slx

SDD_SF_event.slx

SDD_SF_exit_substate.slx

SDD_SF_model.slx


SDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors

SDD_blkdiagSDD_execorder

SDD_nameSDD_path

SDD_chksumSDD_SF_chartSDD_SFstatesSDD_SFtransSDD_SFevents

SDD_SFgraphfuncSDD_SFdata

SDD_SFtargetSDD_configsetSDD_rptformat_PDFSDD_rptformat_HTML

SDD_rptformat_DOCSDD_rptformat_RTF


reports


15/25

3-5


SDD_subsys.slx SDD_mdlver


SDD_blkdiagSDD_inputs

SDD_outputsSDD_blkparasSDD_execorder

SDD_nameSDD_pathSDD_chksumSDD_configset

SDD_rptformat_PDFSDD_rptformat_HTML


Included in


SDD_truth_table_model.slx

SDD_TT_Block.slxSDD_mdlver

SDD_checksumSDD_userdef_titleSDD_userdef_subtitle

SDD_userdef_authorsSDD_blkdiagSDD_execorder

SDD_nameSDD_path

SDD_SF_chartSDD_SFtt

SDD_SFdataSDD_SFtargetSDD_configsetSDD_rptformat_PDFSDD_rptformat_HTML


Included in



16/25

3-6


SDD_Version_Description_Test.slx SDD_mdlver


SDD_blkdiagSDD_descr

SDD_inputsSDD_outputsSDD_configset


Included in


3.2 Model XML Comparison Report –

TraceabilityMatrixBi-directional traceability between the Tool Operation Requirements for the Model XMLComparison Report capability and test cases, procedures and associated test files is provided in

the traceability matrix:

qualkitdo_xmlcomp_trace.xlsx

The traceability matrix is located in:

matlabroot/toolbox/qualkits/do/rptgenext/


17/25

4 Executing Test Cases andProcedures

4.1 Executing Test Cases and Procedures for System

Design Description ReportThe applicant will execute the test cases and procedures for the System Design Description(SDD) report capability in the installed environment.

Note:

To run the test cases and procedures, copy the

matlabroot/toolbox/qualkits/do/rptgenext folder and subfolders to alocation where you have write access, and then run the test cases and procedures fromthere.

Before you execute the RPT file, start a new MATLAB® session.

To execute the RPT file, you must have MATLAB® Report Generator™ and Simulink ®

Report Generator™ licenses.

To execute some test models, you must have Stateflow® and Simulink ® Verification and

Validation™ licenses.

Execute the System Design Description Report test procedure:

1. Configure the report generation:

a. Specify the model covered in the report using thequalkitdoCtrl.rptgenext.model variable in the base workspace:

At the MATLAB command line, enter the following command, where

model_file is the name of the test procedure:


18/25

4-2

qualkitdoCtrl.rptgenext.model = {'model_file'};

b. Specify the report format you want using the

qualkitdoCtrl.rptgenext.reportFormat variable in the baseworkspace:

At the MATLAB command line, enter

qualkitdoCtrl.rptgenext.reportFormat = {'PDF'}; togenerate a report in PDF format

qualkitdoCtrl.rptgenext.reportFormat = {'HTML'}; togenerate a report in HTML format

qualkitdoCtrl.rptgenext.reportFormat = {'DOC'}; togenerate a report in DOC format

qualkitdoCtrl.rptgenext.reportFormat = {'RTF'}; to

generate a report in RTF format

2. Create the report:

Using the GUI:

a. Double-click the qualkitdo_rptgenext_tcp.rpt file to open theReport Explorer.

b. Select File > Report. Simulink Report Generator creates the summaryreport and the model level reports.


report ('qualkitdo_rptgenext_tcp')

3. After the report generation is complete, review the test report results. For moreinformation, see “Test Results”.


19/25

4-3

4.2 Executing Test Cases and Procedures for ModelXML Comparison ReportThe applicant will execute the test cases and procedures for the Model XML Comparison report

capabilities in the installed environment.

Note:

To run the test cases and procedures, copy the

matlabroot/toolbox/qualkits/do/rptgenext folder and subfolders to a

location where you have write access, and then run the test cases and procedures fromthere.

Before you execute the RPT file, start a new MATLAB® session.

Set the Java™ heap size for your MATLAB session to at least 3072 MB before you

execute model coverage RPT files. To check the Java heap size, open the MATLABPreferences dialog box and select General > Java Heap Memory. If the Java Heap Sizevalue is less than 3072 MB, change it to 3072 MB, click OK, and restart MATLAB. (If the

maximum available heap size value is less than 3072 MB, select the maximum value.) This

may help you avoid java.lang.OutOfMemoryError messages.

To execute the RPT file, you must have MATLAB® Report Generator™ and Simulink ®

Report Generator™ licenses.

To execute some test models, you must have Stateflow® and Simulink ® Verification and

Validation™ licenses.

Execute the Model XML Comparison Report test procedure:

1. If you need to qualify the operation of the comparison tool on models saved in previousreleases, you may specify two Simulink releases to test.

At the MATLAB command line, enterqualkitdoCtrl.xmlcomp.mdlFormat = {'FORMAT1', 'FORMAT2'}

where FORMAT1 and FORMAT2 specify one of the following Simulink releases: 'R14', 'R14SP1', 'R14SP2', 'R14SP3', 'R2006A', 'R2006B','R2007A', 'R2007B', 'R2008A', 'R2008B', 'R2009A','R2009B','R2010A', 'R2010B', 'R2011A', 'R2011B', 'R2012A','R2012A_MDL', 'R2012A_SLX', 'R2012B', 'R2012B_MDL','R2012B_SLX', 'R2013A', 'R2013A_MDL', 'R2013A_SLX',


20/25

4-4

'R2013B', 'R2013B_MDL', 'R2013B_SLX', 'R2014A','R2014A_MDL', 'R2014A_SLX', 'R2014B', 'R2014B_MDL','R2014B_SLX', 'R2015B'

Examples of use:

- To qualify the comparison of models saved in R2012b and R2015b, enter:

qualkitdoCtrl.xmlcomp.mdlFormat = {'R2012B','R2015B'}

- To qualify the comparison of models from release R2012b to R2014a, enter:

qualkitdoCtrl.xmlcomp.mdlFormat = {'R2012B','R2014A'}

- If Models are saved in the current release,qualkitdoCtrl.xmlcomp.mdlFormat does not need to be set. It willdefault to the current release.

2. Run the RPT file:

Using the GUI:

Double-click the qualkitdo_xmlcomp_rs.rpt file to open the ReportExplorer

Select File > Report. Simulink Report Generator creates the summary report

and the model level reports


report ('qualkitdo_xmlcomp_rs.rpt')

Simulink Report Generator runs the Model XML Comparison Report tests and stores the

comparison reports and summary qualification report in this generated folder:

rptgenext/outputs/

The summary qualification report

qualkitdo_xmlcomp_qualificationreport.html contains information abouttest environment, executed test files, and qualification results.


21/25

4-5

3. After Simulink Report Generator has finished running the test procedure, verify that thereare no warning or error messages in the Simulink Report Generator message list window.

4. Review the summary qualification report. For more information, see Test Results.


22/25

5 Test Results


23/25

5-2

5.1 System Design Description ReportExecuting the following test procedure generates the following test result files. The test result

files provide the Tool Verification Results in a summary report and multiple model levelreports.

System Design Description Report — Test Summary

Test Procedure Test Results

qualkitdo_rptgenex

t_tcp.rpt1 outputs\qualkitdo_rptgenext_qualificationreport _for_.pdf2

outputs\qualkitdo_rptgenext_qualificationreport _model.pdf3

outputs\qualkitdo_rptgenext_qualificationreport _model.doc3

outputs\qualkitdo_rptgenext_qualificationreport _model.rtf3

outputs\qualkitdo_rptgenext_qualificationreport _model.html3

Notes: 1 Requires MATLAB® Report Generator™, Simulink ® Report Generator™, Simulink ®, Stateflow®, andSimulink ® Verification and Validation™ licenses. 2 Overview report.3 Model level report.

In the above listed test results, model stands forslvnvdemo_fuelsys_docreq, slvnvdemo_fuelsys_htmreq, sldemo_mdlref_basic,

SDD_cont, SDD_discont, SDD_discrete, SDD_eml_call_gf, SDD_logic, SDD_lookup,SDD_math, SDD_other, SDD_SF_ArithLogic, SDD_SF_data_check, SDD_SF_event,SDD_SF_exit_substate, SDD_SF_model, SDD_signals, SDD_sources, SDD_subsys,SDD_truth_table_model, SDD_TT_Block, SDD_Version_Description_Test.

The applicant will:

Review the Tool Verification test results.

Confirm that test reports are generated without errors or warnings.

Configure the test results in a configuration management system.

For each test result file, the DO Qualification Kit for Simulink ® Report Generator™ contains a

pre-generated version of the reports with reviewed content. For each test result file, compare theexpected results (pre-generated version) to the actual results (locally generated version).

Perform this review only on the file formats that will be used for the specific project for whichthe Simulink Report Generator is to be qualified.

When reviewing the actual results (locally generated version) in the qualification reports, theStateflow IDs for unlabeled transitions can differ from those provided in the expected results(pre-generated version). However, the reports -generate the traceability information, even for

models using unlabeled Stateflow transitions. The SDD report capability of Simulink ReportGenerator generates traceability information for a labeled Stateflow transition using the

transition label for the name of the transition. For example, [fails>=5]. In the case of


24/25

5-3

unlabeled transitions, however, SDD uses a string with the word Transition followed by an

ID. The ID is the value of the ID property of the Stateflow transition object,

Stateflow.Transition. Since the value of the ID property for a transition object changes

every time the model is opened, regenerating the report for the same model might lead todifferent transition names. The transition names might not match those provided in pre-

generated version of the report.

The actual model checksums (locally generated version) in the qualification reports can differ

from the checksums provided in the reference reports (pre-generated version). The locallygenerated version of the model checksums depend on the execution environment and can be

platform-specific. If the execution platform of the locally generated version is not the same as

the execution platform used to generated the reference reports (pre-generated version), themodel checksums can differ.


25/25

5-4

5.2 Model XML Comparison Report

Executing the aggregate test procedure file qualkitdo_xmlcomp_rs.rpt generates and

stores the following data in /outputs/:

Generated summary qualification reportqualkitdo_xmlcomp_qualificationreport.html

Generated Model XML Comparison reports

Note: Summary qualification report will include lists of the parameters that were tested

in the appendixes.

The applicant will:

1. Review the test results:

Review the summary qualification report

qualkitdo_xmlcomp_qualificationreport.html , generated whenqualkitdo_xmlcomp_rs.rpt executes.

- Check that the information regarding test environment is correct.

- In the “Qualification Test Results” section of the report, check that thequalification test outcome is evaluated as PASS.

Note 1: If the qualification status is evaluated as FAIL, manually review thecontent of the comparison reports for parameters evaluated as FAILED to assess

the impact of the failed parameter with respect to the software life cycle activities

performed using the tool. The status for each parameter is provided in theappendices of the summary qualification report.

Note 2: A list of corresponding test cases is provided in the CMP page of the

traceability matrix qualkitdo_xmlcomp_trace.xlsx .

2. Configure the content of the folder comprising the test cases, procedures and results in aconfiguration management system.

Documents

Qualkitdo Rptgenext Tcp