MBSE for System Testing

Ron Kratzke, Vitech [email protected]

MBSE for System TestingManaging the development of system testing using the principles of Model Based System Engineering

mailto:[email protected]

Vitech Corporation 2

Context Diagram - Verification Process(Ref: INCOSE Handbook Ver 3.2.2 Oct 2011)

Controls - Applicable Laws and Regulations - Industry Standards - Agreements - Project Procedures and Standards -Project Directives

Outputs - Verification Strategy - Verification Enabling System Requirements - Verification Constraints on Design - Verification Procedures -Final RVTM - Verified System - Verification Report

Enablers - Organization / Enterprise Policies, Practices, and Procedures - Organization / Enterprise Infrastructure - Project Infrastructure - Verification Enabling Systems

Activities - Plan Verification - Perform Verification

Inputs - System Requirements - Verification Criteria - Specification Tree - Updated RVTM - Interface Requirements - Integrated System - Interface Control Documents - Integration Report


Model-Based Systems Engineering

• Model-based systems engineering (MBSE) is the formalized application of modeling to support system requirements, design, analysis, verification and validation activities beginning in the conceptual design phase and continuing throughout development and later life cycle phases.

– From “INCOSE Systems Engineering Vision” 2020 INCOSE-TP-2004-004-02 September, 2007


MODELS

VITECH CORPORATION 5

Everyone has a model …


System Testing has to account for a number of items…

Requirements

Functions(Behavior Architecture)

Components(Physical Architecture)

Testing


MBSE Verification Process ModelMust …

• Provides insight into the problem and potential solutions

• Provide order and traceability• Possess integrity and consistency• Have ability demonstrate and explain the test

program


Is The VCRM Enough???Number Name Description method

R.1.1.1 Availability The system shall be unavailable no more than a total of ten minutes per month.

Analysis

R.2.1.1.1.1 Media of Requests: Hardcopy Forms

The system shall accept requests via hardcopy forms. Demonstration

R.2.1.1.1.4 Media of Requests: Telephonic Electronic File

The system shall accept requests via telephonic electronic file. Test

R.2.1.1.1.5 Media of Requests: Web Services The system shall accept requests via via a Web service. Test R.2.2.2 Provide Products The system shall provides previously collected images/products, if

appropriate. Demonstration

R.2.3.1 Control Multiple Collectors The system shall control multiple image collectors. Test R.2.3.1 Control Multiple Collectors The system shall control multiple image collectors. Test R.2.3.1 Control Multiple Collectors The system shall control multiple image collectors. Demonstration FN.1.1 Accept And Format Request The system shall accept the requests for information, verify that the

requester is a valid customer of the system, and format the request into a form and media that the system can use.

Test

FN.1.1 Accept And Format Request -Without Certification

The system shall accept the requests for information and format the request into a form and media that the system can use.

Test

FN.2.1 Check Product Inventory The system shall check the imagery product inventory to determine if the customer's request for imagery products is available from current imagery product inventory.

Test

FN.2.2 Prioritize Request The system shall organize customer orders using a priority based approach. Priorities shall be assigned based upon delivery date, customer need, current collection assignments, and system resources.

Test

FN.2.3 Determine Collector Mix The system shall evaluate the customer's order to establish which set of collectors shall best provide imagery products answering the customer's need.

Test

FN.2.4 Notify User Of Estimated Schedule The system shall prepare and send to the customer a notification estimating the delivery date of the customer's order.

Test


Additional Information Needed

• In addition to the method, shouldn’t we also know?– Objective– Environment– Special Conditions– Success Criteria

Ref: A Day in the Life of a Verification Requirement Tutorial, Northrop Grumman, Steve Scukanec, 27th Annual T&E Conference, March 14, 2011


Status of Completing the Verification

• Wouldn’t we also want to track?– Method (which is in the VCRM)– Planning Status – Completion Status– Where / When the Verification Event will take

place


Verification Requirement Statement

Verification Requirement Description statement provides the details on HOW the verification will be accomplished


Relating Requirements to Verification

Function,Requirement

verified by(verifies)

VerificationRequirement

Verification Requirement attributes: - Description - Method - Status - Level

Add two more attributes


Table Output from Model …Number Name VR No. VR Name method Status Level of Test

R.1.1.1 Availability V.R.1.1 Continuous Support Analysis Planned System R.2.1.1.1.1 Media of Requests: Hardcopy

Forms V.R.2.1.1.1 Media of Requests: Hardcopy

Forms Demonstration Planned Subsystem

R.2.1.1.1.4 Media of Requests: Telephonic Electronic File

V.R.2.1.1.4 Media of Requests: Telephone Electronic File

Test In Progress Subsystem

R.2.1.1.1.5 Media of Requests: Web Services

V.R.2.1.1.5 Media of Requests: Web-based Electronic File

Test Not Yet Planned

Subsystem

R.2.2.2 Provide Products V.R.2.2.2 Provide Products Demonstration Not Yet Planned

System

R.2.3.1 Control Multiple Collectors V.R.2.3.1.a Control Multiple Collectors - Laboratory

Test Planned Subsystem

R.2.3.1 Control Multiple Collectors V.R.2.3.1.b Control Multiple Collectors - Ground Testing

Test Completed - Unsatisfactory

Subsystem

R.2.3.1 Control Multiple Collectors V.R.2.3.1.c Control Multiple Collectors - Flight Testing

Demonstration In Progress Subsystem

FN.1.1 Accept And Format Request V.R.2.5 Provide Feedback Test Completed - Satisfactory

System

FN.1.1 Accept And Format Request -Without Certification

V.R.2.5 Provide Feedback Test Completed - Satisfactory

System

FN.2.1 Check Product Inventory V.R.2.5 Provide Feedback Test Completed - Satisfactory

System

FN.2.2 Prioritize Request V.R.2.5 Provide Feedback Test Completed - Satisfactory

System

FN.2.3 Determine Collector Mix V.R.2.5 Provide Feedback Test Completed - Satisfactory

System

FN.2.4 Notify User Of Estimated Schedule

V.R.2.5 Provide Feedback Test Completed - Satisfactory

System

Provides the data from the model to provide overall status of verification planning… But what about the test events?


What about the actual Testing?Function,

Requirement



What test events are planned?Which Verification Requirements will be verified in which test event?Who is doing the testing? What Organization is responsible?What configuration is going to be used in testing?Which Test Procedures will be used during any particular Test Event?



Requirement



VerificationEventfulfilled by

(fulfills)

Add “Verification Event” to specify and describe a Test Event which will be used to verify a subset of Verification Requirements (for example, Limited User Test, Laboratory Bench Test, Operational Test, etc.)



Requirement



VerificationEvent

Organization

TestProcedure

assigned to(responsible for)

employs(employed by)

TestConfiguration


fulfilled by(fulfills)

Relate the Verification Event to an Organization, Test Configuration, and Test Procedure


Provide a complete picture…


Verification Event InformationVerification Event Test Organization Test Config/Procedure Name

Limited System Functional Test Acme Test Labs TestConfiguration Test Database TestProcedure Data Base Access TestProcedure Data Entry Procedure

Logistic Test Event Logistics Analysts, Inc TestConfiguration Forward Operation Configuration TestConfiguration Permanent Cmd Ctr Configuration TestProcedure 30 Day System Operation Model

Media Usage Testing DataBase Testers R Us TestConfiguration Forward Operation Configuration TestConfiguration Test Database TestProcedure Media Usage Test Procedure

Multiple Collector Testing Collector Image Management, LLC TestConfiguration Global Hawk UAV Test Interface TestConfiguration Micro UAV Test Interface TestProcedure Two Collector Tasking Procedure


What about time and resources for the Verification Requirement?

Function,Requirement



VerificationEvent

Organization

TestProcedure

assigned to(responsible for)


TestConfiguration


fulfilled by(fulfills)

Test Activity

executed by(executes)


Test Activity – Detailed Test Planning


Produce a Test Plan from the Model


Summary …• System Testing model

– Extends traditional VCRM to provide added insight– Plan Verification Events including support resources– Manage Testing Activity sequence and timing– Produce Test Plans

• Traceability• Completeness• Consistency• Document Generation

Ron Kratzke, Vitech [email protected]

MBSE for System TestingQuestions ???

mailto:[email protected]