8/20/2019 Qualkitdo Rptgenext Tcp
1/25
DO Qualification Kit
Simulink® Report Generator ™ Test Cases and Procedures
R2015b, September 2015
8/20/2019 Qualkitdo Rptgenext Tcp
2/25
How to Contact MathWorks
Latest news: www.mathworks.com
Sales and services: www.mathworks.com/sales_and_services
User community: www.mathworks.com/matlabcentral
Technical support: www.mathworks.com/support/contact_us
Phone: 508-647-7000
The MathWorks, Inc.
3 Apple Hill Drive Natick, MA 01760-2098
DO Qualification Kit: Simulink ® Report Generator ™ Test Cases and Procedures
© COPYRIGHT 2010 – 2015 by The MathWorks, Inc.
The software described in this document is furnished under a license agreement. The software may be used or copied only under
the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written
consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the
federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees
that this software or documentation qualifies as commercial computer software or commercial computer software documentation
as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms andconditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,
reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or
other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.
If this License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks f or alist of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective
holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents f or moreinformation.
http://www.mathworks.com/http://www.mathworks.com/http://www.mathworks.com/sales_and_serviceshttp://www.mathworks.com/sales_and_serviceshttp://www.mathworks.com/matlabcentral/http://www.mathworks.com/matlabcentral/http://www.mathworks.com/support/contact_us/http://www.mathworks.com/support/contact_us/http://www.mathworks.com/trademarkshttp://www.mathworks.com/trademarkshttp://www.mathworks.com/trademarkshttp://www.mathworks.com/patentshttp://www.mathworks.com/patentshttp://www.mathworks.com/patentshttp://www.mathworks.com/patentshttp://www.mathworks.com/trademarkshttp://www.mathworks.com/support/contact_us/http://www.mathworks.com/matlabcentral/http://www.mathworks.com/sales_and_serviceshttp://www.mathworks.com/
8/20/2019 Qualkitdo Rptgenext Tcp
3/25
Revision History
September 2010 New for Version 1.3 (Applies to Release 2010b)
April 2011 Revised for Version 1.4 (Applies to Release 2011a)September 2011 Revised for Version 1.5 (Applies to Release 2011b)
March 2012 Revised for Version 1.6 (Applies to Release 2012a)
September 2012 Revised for Version 2.0 (Applies to Release 2012b)
March 2013 Revised for Version 2.1 (Applies to Release 2013a)
September 2013 Revised for Version 2.2 (Applies to Release 2013b)
March 2014 Revised for Version 2.3 (Applies to Release 2014a)
October 2014 Revised for Version 2.4 (Applies to Release 2014b)
March 2015 Revised for Version 2.5 (Applies to Release 2015a)
September 2015 Revised for DO Qualification Kit Version 3.0 (Applies to Release 2015b); renamed to Simulink ®
Report Generator™ Test Cases and Procedures
8/20/2019 Qualkitdo Rptgenext Tcp
4/25
8/20/2019 Qualkitdo Rptgenext Tcp
5/25
v
Contents
1
Introduction ...................................................................................................................................... 1-1
2
Test Cases and Procedures ............................................................................................................... 2-1
2.1
System Design Description Report ......................................................................................... 2-1
2.2
Model XML Comparison Report ............................................................................................ 2-1
3
Traceability Between Operational Requirements and Test Cases and Procedures ........................... 3-1
3.1
System Design Description Report – Traceability Matrix....................................................... 3-2
3.2
Model XML Comparison Report – Traceability Matrix ......................................................... 3-6
4
Executing Test Cases and Procedures .............................................................................................. 4-1
4.1
Executing Test Cases and Procedures for System Design Description Report ....................... 4-1
4.2
Executing Test Cases and Procedures for Model XML Comparison Report .......................... 4-3
5 Test Results ...................................................................................................................................... 5-1
5.1 System Design Description Report ......................................................................................... 5-2 5.2
Model XML Comparison Report ............................................................................................ 5-4
8/20/2019 Qualkitdo Rptgenext Tcp
6/25
vi
8/20/2019 Qualkitdo Rptgenext Tcp
7/25
1 Introduction
This document comprises test cases and procedures (Reference DO-330 Sections 10.3.3 and
10.3.4) for the following capability of the Simulink ® Report Generator ™ verification tool:
System Design Description report
Model XML Comparison report
This document is intended for use in the DO-330 tool qualification process for TQL-5 tools.
See also the DO Qualification Kit User's Guide, R2015b.
8/20/2019 Qualkitdo Rptgenext Tcp
8/25
1-2
8/20/2019 Qualkitdo Rptgenext Tcp
9/25
2 Test Cases and Procedures
Requirements- based test cases and procedures have been developed from the “System Design
Description Report Operational Requirements” and “Model XML Comparison Report
Operational Requirements” sections in the following document:
Simulink Report Generator Tool Operational Requirements
To access the tool operational requirements document, on the MATLAB® command line, type
qualkitdo to open the Artifacts Explorer. The document is in Simulink Report Generator.
2.1 System Design Description ReportThe test cases and procedures exercise the System Design Description (SDD) report capability,
and reside in the Simulink ® test models identified in System Design Description Report – Traceability Matrix. The test procedure applies each SDD report capability to a subset of the
preceding test models, and resides in the report generator file identified in System Design
Description Report – Traceability Matrix.
2.2 Model XML Comparison ReportThe test cases and procedures exercising the Model XML Comparison report capability include
Simulink ® Report Generator setup file qualkitdo_xmlcomp_rs.rpt and associated m-
files, mat-files, and Simulink test models. Test files and their allocation to the tool operationalrequirements are identified in Model XML Comparison Report – Traceability Matrix.
The report setup file invokes MATLAB code that sequentially modifies values of each
qualifying parameter of the test model using pre-generated test values and runs XMLcomparison between the original and modified model. Generated comparison reports areevaluated to confirm that the reports contain the correct information for the corresponding
parameters. Qualification status is assigned to PASS if all generated reports are correct. The
summary qualification test report summarizes the outcome of the tests run and include lists of
8/20/2019 Qualkitdo Rptgenext Tcp
10/25
2-2
the qualified parameters in the appendixes. Figure 1 shows the Model XML Comparison testing
workflow.
Figure 1: Test Workflow
8/20/2019 Qualkitdo Rptgenext Tcp
11/25
3 Traceability Between OperationalRequirements and Test Casesand Procedures
8/20/2019 Qualkitdo Rptgenext Tcp
12/25
3-2
3.1 System Design Description Report – TraceabilityMatrixThe test cases and procedures cover the Tool Operational Requirements for the System Design
Description (SDD) report, as shown in the following table. The test models are used by the test
procedure qualkitdo_rptgenext_tcp.rpt .
System Design Description Report – Traceability Matrix
Test Model Requirement ID ExpectedResult
slvnvdemo_fuelsys_docreq.slxslvnvdemo_fuelsys_htmreq.slx
SDD_mdlverSDD_checksum
SDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authorsSDD_blkdiag
SDD_execorderSDD_name
SDD_path
SDD_SF_chartSDD_dataSDD_SFtarget
SDD_reqsSDD_configset
SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOC
SDD_rptformat_RTF
Included in pre-generated
reports
sldemo_mdlref_basic.slx SDD_mdlver
SDD_checksumSDD_userdef_titleSDD_userdef_subtitle
SDD_userdef_authorsSDD_blkdiagSDD_blkparasSDD_execorder
SDD_mdlrefSDD_configset
SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOCSDD_rptformat_RTF
Included in
pre-generatedreports
8/20/2019 Qualkitdo Rptgenext Tcp
13/25
3-3
Test Model Requirement ID ExpectedResult
SDD_cont.slx
SDD_discont.slxSDD_discrete.slx
SDD_logic.slx
SDD_lookup.slx
SDD_math.slx
SDD_other.slx
SDD_signals.slx
SDD_sources.slx
SDD_mdlverSDD_checksum
SDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors
SDD_blkdiagSDD_blkparas
SDD_mdlverSDD_checksumSDD_userdef_title
SDD_userdef_subtitleSDD_userdef_authorsSDD_blkdiagSDD_data
SDD_nameSDD_path
SDD_chksumSDD_SF_chartSDD_SFttSDD_eml
SDD_SFtargetSDD_configset
SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOC
SDD_rptformat_RTF
Included in pre-generated
reports
8/20/2019 Qualkitdo Rptgenext Tcp
14/25
3-4
Test Model Requirement ID ExpectedResult
SDD_eml_call_gf.slx SDD_mdlver
SDD_checksumSDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors
SDD_blkdiagSDD_outputs
SDD_execorderSDD_nameSDD_path
SDD_SF_chartSDD_SFgraphfuncSDD_SFdataSDD_eml
SDD_SFtargetSDD_configset
SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOCSDD_rptformat_RTF
Included in
pre-generatedreports
SDD_SF_ArithLogic.slx
SDD_SF_data_check.slx
SDD_SF_event.slx
SDD_SF_exit_substate.slx
SDD_SF_model.slx
SDD_mdlverSDD_checksum
SDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors
SDD_blkdiagSDD_execorder
SDD_nameSDD_path
SDD_chksumSDD_SF_chartSDD_SFstatesSDD_SFtransSDD_SFevents
SDD_SFgraphfuncSDD_SFdata
SDD_SFtargetSDD_configsetSDD_rptformat_PDFSDD_rptformat_HTML
SDD_rptformat_DOCSDD_rptformat_RTF
Included in pre-generated
reports
8/20/2019 Qualkitdo Rptgenext Tcp
15/25
3-5
Test Model Requirement ID ExpectedResult
SDD_subsys.slx SDD_mdlver
SDD_checksumSDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors
SDD_blkdiagSDD_inputs
SDD_outputsSDD_blkparasSDD_execorder
SDD_nameSDD_pathSDD_chksumSDD_configset
SDD_rptformat_PDFSDD_rptformat_HTML
SDD_rptformat_DOCSDD_rptformat_RTF
Included in
pre-generatedreports
SDD_truth_table_model.slx
SDD_TT_Block.slxSDD_mdlver
SDD_checksumSDD_userdef_titleSDD_userdef_subtitle
SDD_userdef_authorsSDD_blkdiagSDD_execorder
SDD_nameSDD_path
SDD_SF_chartSDD_SFtt
SDD_SFdataSDD_SFtargetSDD_configsetSDD_rptformat_PDFSDD_rptformat_HTML
SDD_rptformat_DOCSDD_rptformat_RTF
Included in
pre-generatedreports
8/20/2019 Qualkitdo Rptgenext Tcp
16/25
3-6
Test Model Requirement ID ExpectedResult
SDD_Version_Description_Test.slx SDD_mdlver
SDD_checksumSDD_userdef_titleSDD_userdef_subtitleSDD_userdef_authors
SDD_blkdiagSDD_descr
SDD_inputsSDD_outputsSDD_configset
SDD_rptformat_PDFSDD_rptformat_HTMLSDD_rptformat_DOCSDD_rptformat_RTF
Included in
pre-generatedreports
3.2 Model XML Comparison Report –
TraceabilityMatrixBi-directional traceability between the Tool Operation Requirements for the Model XMLComparison Report capability and test cases, procedures and associated test files is provided in
the traceability matrix:
qualkitdo_xmlcomp_trace.xlsx
The traceability matrix is located in:
matlabroot/toolbox/qualkits/do/rptgenext/
8/20/2019 Qualkitdo Rptgenext Tcp
17/25
4 Executing Test Cases andProcedures
4.1 Executing Test Cases and Procedures for System
Design Description ReportThe applicant will execute the test cases and procedures for the System Design Description(SDD) report capability in the installed environment.
Note:
To run the test cases and procedures, copy the
matlabroot/toolbox/qualkits/do/rptgenext folder and subfolders to alocation where you have write access, and then run the test cases and procedures fromthere.
Before you execute the RPT file, start a new MATLAB® session.
To execute the RPT file, you must have MATLAB® Report Generator™ and Simulink ®
Report Generator™ licenses.
To execute some test models, you must have Stateflow® and Simulink ® Verification and
Validation™ licenses.
Execute the System Design Description Report test procedure:
1. Configure the report generation:
a. Specify the model covered in the report using thequalkitdoCtrl.rptgenext.model variable in the base workspace:
At the MATLAB command line, enter the following command, where
model_file is the name of the test procedure:
8/20/2019 Qualkitdo Rptgenext Tcp
18/25
4-2
qualkitdoCtrl.rptgenext.model = {'model_file'};
b. Specify the report format you want using the
qualkitdoCtrl.rptgenext.reportFormat variable in the baseworkspace:
At the MATLAB command line, enter
qualkitdoCtrl.rptgenext.reportFormat = {'PDF'}; togenerate a report in PDF format
qualkitdoCtrl.rptgenext.reportFormat = {'HTML'}; togenerate a report in HTML format
qualkitdoCtrl.rptgenext.reportFormat = {'DOC'}; togenerate a report in DOC format
qualkitdoCtrl.rptgenext.reportFormat = {'RTF'}; to
generate a report in RTF format
2. Create the report:
Using the GUI:
a. Double-click the qualkitdo_rptgenext_tcp.rpt file to open theReport Explorer.
b. Select File > Report. Simulink Report Generator creates the summaryreport and the model level reports.
At the MATLAB command line, enter
report ('qualkitdo_rptgenext_tcp')
3. After the report generation is complete, review the test report results. For moreinformation, see “Test Results”.
8/20/2019 Qualkitdo Rptgenext Tcp
19/25
4-3
4.2 Executing Test Cases and Procedures for ModelXML Comparison ReportThe applicant will execute the test cases and procedures for the Model XML Comparison report
capabilities in the installed environment.
Note:
To run the test cases and procedures, copy the
matlabroot/toolbox/qualkits/do/rptgenext folder and subfolders to a
location where you have write access, and then run the test cases and procedures fromthere.
Before you execute the RPT file, start a new MATLAB® session.
Set the Java™ heap size for your MATLAB session to at least 3072 MB before you
execute model coverage RPT files. To check the Java heap size, open the MATLABPreferences dialog box and select General > Java Heap Memory. If the Java Heap Sizevalue is less than 3072 MB, change it to 3072 MB, click OK, and restart MATLAB. (If the
maximum available heap size value is less than 3072 MB, select the maximum value.) This
may help you avoid java.lang.OutOfMemoryError messages.
To execute the RPT file, you must have MATLAB® Report Generator™ and Simulink ®
Report Generator™ licenses.
To execute some test models, you must have Stateflow® and Simulink ® Verification and
Validation™ licenses.
Execute the Model XML Comparison Report test procedure:
1. If you need to qualify the operation of the comparison tool on models saved in previousreleases, you may specify two Simulink releases to test.
At the MATLAB command line, enterqualkitdoCtrl.xmlcomp.mdlFormat = {'FORMAT1', 'FORMAT2'}
where FORMAT1 and FORMAT2 specify one of the following Simulink releases: 'R14', 'R14SP1', 'R14SP2', 'R14SP3', 'R2006A', 'R2006B','R2007A', 'R2007B', 'R2008A', 'R2008B', 'R2009A','R2009B','R2010A', 'R2010B', 'R2011A', 'R2011B', 'R2012A','R2012A_MDL', 'R2012A_SLX', 'R2012B', 'R2012B_MDL','R2012B_SLX', 'R2013A', 'R2013A_MDL', 'R2013A_SLX',
8/20/2019 Qualkitdo Rptgenext Tcp
20/25
4-4
'R2013B', 'R2013B_MDL', 'R2013B_SLX', 'R2014A','R2014A_MDL', 'R2014A_SLX', 'R2014B', 'R2014B_MDL','R2014B_SLX', 'R2015B'
Examples of use:
- To qualify the comparison of models saved in R2012b and R2015b, enter:
qualkitdoCtrl.xmlcomp.mdlFormat = {'R2012B','R2015B'}
- To qualify the comparison of models from release R2012b to R2014a, enter:
qualkitdoCtrl.xmlcomp.mdlFormat = {'R2012B','R2014A'}
- If Models are saved in the current release,qualkitdoCtrl.xmlcomp.mdlFormat does not need to be set. It willdefault to the current release.
2. Run the RPT file:
Using the GUI:
Double-click the qualkitdo_xmlcomp_rs.rpt file to open the ReportExplorer
Select File > Report. Simulink Report Generator creates the summary report
and the model level reports
At the MATLAB command line, enter
report ('qualkitdo_xmlcomp_rs.rpt')
Simulink Report Generator runs the Model XML Comparison Report tests and stores the
comparison reports and summary qualification report in this generated folder:
rptgenext/outputs/
The summary qualification report
qualkitdo_xmlcomp_qualificationreport.html contains information abouttest environment, executed test files, and qualification results.
8/20/2019 Qualkitdo Rptgenext Tcp
21/25
4-5
3. After Simulink Report Generator has finished running the test procedure, verify that thereare no warning or error messages in the Simulink Report Generator message list window.
4. Review the summary qualification report. For more information, see Test Results.
8/20/2019 Qualkitdo Rptgenext Tcp
22/25
5 Test Results
8/20/2019 Qualkitdo Rptgenext Tcp
23/25
5-2
5.1 System Design Description ReportExecuting the following test procedure generates the following test result files. The test result
files provide the Tool Verification Results in a summary report and multiple model levelreports.
System Design Description Report — Test Summary
Test Procedure Test Results
qualkitdo_rptgenex
t_tcp.rpt1 outputs\qualkitdo_rptgenext_qualificationreport _for_.pdf2
outputs\qualkitdo_rptgenext_qualificationreport _model.pdf3
outputs\qualkitdo_rptgenext_qualificationreport _model.doc3
outputs\qualkitdo_rptgenext_qualificationreport _model.rtf3
outputs\qualkitdo_rptgenext_qualificationreport _model.html3
Notes: 1 Requires MATLAB® Report Generator™, Simulink ® Report Generator™, Simulink ®, Stateflow®, andSimulink ® Verification and Validation™ licenses. 2 Overview report.3 Model level report.
In the above listed test results, model stands forslvnvdemo_fuelsys_docreq, slvnvdemo_fuelsys_htmreq, sldemo_mdlref_basic,
SDD_cont, SDD_discont, SDD_discrete, SDD_eml_call_gf, SDD_logic, SDD_lookup,SDD_math, SDD_other, SDD_SF_ArithLogic, SDD_SF_data_check, SDD_SF_event,SDD_SF_exit_substate, SDD_SF_model, SDD_signals, SDD_sources, SDD_subsys,SDD_truth_table_model, SDD_TT_Block, SDD_Version_Description_Test.
The applicant will:
Review the Tool Verification test results.
Confirm that test reports are generated without errors or warnings.
Configure the test results in a configuration management system.
For each test result file, the DO Qualification Kit for Simulink ® Report Generator™ contains a
pre-generated version of the reports with reviewed content. For each test result file, compare theexpected results (pre-generated version) to the actual results (locally generated version).
Perform this review only on the file formats that will be used for the specific project for whichthe Simulink Report Generator is to be qualified.
When reviewing the actual results (locally generated version) in the qualification reports, theStateflow IDs for unlabeled transitions can differ from those provided in the expected results(pre-generated version). However, the reports -generate the traceability information, even for
models using unlabeled Stateflow transitions. The SDD report capability of Simulink ReportGenerator generates traceability information for a labeled Stateflow transition using the
transition label for the name of the transition. For example, [fails>=5]. In the case of
8/20/2019 Qualkitdo Rptgenext Tcp
24/25
5-3
unlabeled transitions, however, SDD uses a string with the word Transition followed by an
ID. The ID is the value of the ID property of the Stateflow transition object,
Stateflow.Transition. Since the value of the ID property for a transition object changes
every time the model is opened, regenerating the report for the same model might lead todifferent transition names. The transition names might not match those provided in pre-
generated version of the report.
The actual model checksums (locally generated version) in the qualification reports can differ
from the checksums provided in the reference reports (pre-generated version). The locallygenerated version of the model checksums depend on the execution environment and can be
platform-specific. If the execution platform of the locally generated version is not the same as
the execution platform used to generated the reference reports (pre-generated version), themodel checksums can differ.
8/20/2019 Qualkitdo Rptgenext Tcp
25/25
5-4
5.2 Model XML Comparison Report
Executing the aggregate test procedure file qualkitdo_xmlcomp_rs.rpt generates and
stores the following data in /outputs/:
Generated summary qualification reportqualkitdo_xmlcomp_qualificationreport.html
Generated Model XML Comparison reports
Note: Summary qualification report will include lists of the parameters that were tested
in the appendixes.
The applicant will:
1. Review the test results:
Review the summary qualification report
qualkitdo_xmlcomp_qualificationreport.html , generated whenqualkitdo_xmlcomp_rs.rpt executes.
- Check that the information regarding test environment is correct.
- In the “Qualification Test Results” section of the report, check that thequalification test outcome is evaluated as PASS.
Note 1: If the qualification status is evaluated as FAIL, manually review thecontent of the comparison reports for parameters evaluated as FAILED to assess
the impact of the failed parameter with respect to the software life cycle activities
performed using the tool. The status for each parameter is provided in theappendices of the summary qualification report.
Note 2: A list of corresponding test cases is provided in the CMP page of the
traceability matrix qualkitdo_xmlcomp_trace.xlsx .
2. Configure the content of the folder comprising the test cases, procedures and results in aconfiguration management system.