An Analysis of An Integrated Systems Engineering & Test

1

An Analysis of An

Integrated Systems Engineering &

Test & Evaluation Approach

for “Collaborative” System of Systems

Mr. Walter Ott

Ph.D. Candidate

The George Washington University

Washington, D.C.

[email protected]

Bill A. Olson, Ph.D. CSEP-Acq

Professorial Lecturer


Washington, D.C.

[email protected]

Paul Blessner, Ph.D.

Professorial Lecturer


Washington, D.C.

[email protected]

mailto:[email protected]



2

Disclaimer The research, results and conclusions, and

recommendations are the opinion of the author

and should not be construed as an official

position of the United States Department of

Defense, Joint Staff or Science Applications

International Corporation (SAIC).

3

Agenda

• Introduction

• The Problem

• Methodology

• Summary

4

Introduction

5

Source: JP 3-09.3 Close Air Support. (2009). Chairman Joint Chiefs of Staff

Close Air Support Mission

• Close Air Support (CAS): air

action by fixed-wing and rotary-

wing aircraft against hostile

targets that are in close proximity

to friendly forces, and requires

detailed integration of each air

mission with the fire and

movement of those forces.”

• Immediate CAS: occurs outside

the planning cycle; may involve reallocating airborne aircraft.

(U.S. Air Force photo by Tech. Sgt. Michael R. Holzworth)

6

Source: JP 3-09.3 Close Air Support. (2009). Chairman Joint Chiefs of Staff

Close Air Support Mission

• Close Air Support (CAS): air

action by fixed-wing and rotary-

wing aircraft against hostile

targets that are in close proximity

to friendly forces, and requires

detailed integration of each air

mission with the fire and

movement of those forces.”

• Immediate CAS: occurs outside

the planning cycle; may involve reallocating airborne aircraft.


Release Number: 370910

9/17/2010 - SOUTHWEST ASIA -- Coalition airpower integrated with Operation

New Dawn partners in Iraq and the International Security Assistance Force in

Afghanistan in the following operations September 16, according to Combined Air

and Space Operations Center officials here.

Air Operations in Afghanistan

September 16, 2010:

Balocan, RC-W

A U.S. Air Force B-1B aircrew, U.S. Navy F/A-18C pilots and U.S. Air Force A-10

Thunderbolt II pilots provided armed overwatch for friendly forces. F/A-18C pilots

conducted a show of force. The B-1B aircrew release precision-guided munitions

over an enemy target to deter enemy activity. A-10 pilots conducted a show of force

, performed multiple strafing runs, and release multiple precision-guided munitions

over multiple enemy positions to deter enemy activity, Joint terminal attack

controllers deemed the engagements successful.

Source: http://www.afcent.af.mil/news/story_print.asp?id=123222782

7

Systems Engineering

• Historical emphasis on system(s)

• Within the last 10-15 years, rise in:

– Family of Systems (FOS)

– Systems of Systems (SoS)

– Enterprise Architecture (EA)

• New emphasis on

– “User capabilities”

• Resolution of “capability gaps”

• Integration of systems

• “End-to-end” process

• “Net-centric” versus “platform centric”

8

System of Systems: “set or arrangement of systems that results when independent and

useful systems are integrated into a larger system that delivers unique capabilities” Source: Defense Acquisition Guidebook. (2013). Retrieved from https://acc.dau.mil/docs/dag_pdf/dag_complete.pdf.

Types of SoS - Directed: SoS objectives, management,

funding, and authority; systems are subordinated

to a SoS.

- Acknowledged: SoS objectives, management,

funding, and authority; however, systems retain

their own management, funding, and authority in

parallel with the SoS.

- Collaborative: No top down objectives,

management authority, responsibility, or

funding at the SoS level; systems voluntarily

work together to address shared or common

interests.

- Virtual: Like collaborative but systems do not

know each other. Source: Office of the Deputy Under Secretary of Defense for Acquisition and

Technology, Systems and Software Engineering. Systems Engineering Guide for

Systems of Systems, Version 1.0. Washington, DC: ODUSD(A&T)SSE, 2008.

Definition Example Systems Stakeholder

Involvement

Governance

(Management)

Operational Focus

(Operational Environment)

Acquisition Test & Evaluation Boundaries & Interfaces Performance & Behavior

System A functionally,

physically, and/or

behaviorally related

group of regularly

interacting or

interdependent

elements; that group

of elements forming a

unified whole. (3)

Aegis Weapon

System

Clearer set of

stakeholders.

Stakeholders

committed to that

system and play specific

roles in the SE of the

system.(11)

Aligned PM and funding.

Governance of the SE

process is usually

hierarchical.

(11)

Designed and developed to

meet operational objectives.

Mission objectives are

established on a structured

requirements or capability

development process along

with defined concepts of

operation and priorities for

development.(13)

Aligned to ACAT

Milestones, documented

requirements, SE with a

Systems Engineering Plan

(SEP).

Test and Evaluation of the

[entire] system, or

subsystems, is possible.

Focuses on boundaries and

interfaces for the single

system.

Performance of the system to

meet specified performance

objectives.

Collaborative

SoS

The component

systems interact more

or less voluntarily to

fulfill agreed upon

central purposes. The

Internet is a

collaborative system.

The Internet

Engineering Task

Force works out

standards but has no

power to enforce

them. The central

players collectively

decide how to

provide or deny

service, thereby

providing some

means of enforcing

and maintaining

standards.

(5)

May include

communities of

practice (COPs).

Digitally-Aided

Close Air Support

(DACAS); Digitally-

Aided Fire Support

(DAFS); Joint

Personnel

Recovery (JPR)

Multiple levels of

stakeholders. Often

competing interests

between individual

system level, and the

SoS level.(12)

Added levels of complexity

due to management and

funding for both SoS and

systems; SoS has no control

over constituent systems.

Constituent system’s

program managers and

systems engineers

collaborative at the SoS

level to influence system

development that meets

SoS needs.

Governance of SoS SE

process will necessarily

take on a collaborative

nature. (12)

Changes in systems to meet

SoS requirements are based

on agreements and

collaboration.

SoS needs are based on

agreement and collaboration,

not top-down authority from

the SoS manager. (6)

Cross multiple system

lifecycles, involving

legacy systems,

developmental systems,

and technology insertion;

collaboration on

development of SoS

requirements.

Multiple systems at

different stages of life

cycle. Need to expand or

redefine existing SE

processes to accommodate

unique considerations of

individual systems to

address the overall SoS

needs.(13)

Focus on identifying

contributing systems, gain

their participation, and enable

data flow, understanding, and

functionality across the SoS

based on participating systems

concurrence.

Performance across the SoS

that meets SoS user capability

needs as agreed upon by the

participating systems.

System versus Collaborative SoS

9

The Problem Set

10

• Services [continue to] independently develop, field, and manage systems that are not interoperable

in the Systems-of-Systems (SoS) environment

– Stove-piped solutions, lack interoperability

– Require “workarounds” or extensive resources to mitigate interoperability shortfalls

– Problem set magnified when operating in the Coalition/multi-national environment

• Question:

– How does the SE community manage the SoS Engineering challenge to achieve an

interoperable SoS?

• SoS challenge magnified by

– Systems at or approaching end of life-cycle

– Independently evolving “legacy” systems

– New and emerging systems

– COTs and proprietary systems/software

– Rate of technological change

The Problem Set

“Asynchronous” Life Cycles

11

Interoperability

1986 1987

1993 1996 1998 2003 2003

• The interoperability problem is not

new

• However, the pace of technology

and the need to operate in a SoS

environment have increased its

impact

Source: Military Readiness: Lingering Training and Equipment Issues Hamper Air Support of Ground Forces. (2003).

Retrieved from http://www.gao.gov/assets/240/238142.pdf.

2005 2008

2003

12

Interoperability

1986 1987

1993 1996 1998 2003 2003

• The interoperability problem is not

new

• However, the pace of technology

and the need to operate in a SoS

environment have increased its

impact

Military services have not yet achieved DoD’s

goal for ensuring that equipment acquired for

close air support missions is interoperable

and cost-effective.

The digital systems … are not interoperable

across the services

The lack of interoperability across the

services reduces the equipment’s

effectiveness and limits its usefulness.

Source: Military Readiness: Lingering Training and Equipment Issues Hamper Air Support of Ground Forces. (2003).

Retrieved from http://www.gao.gov/assets/240/238142.pdf.

2005 2008

2003

13

Interoperability • Interoperability: The ability of systems, units, or forces to provide data, information,

materiel, and services to and accept the same from other systems, units, or forces AND to

use the data, information, materiel, and services so exchanged to enable them to operate

effectively together. … Interoperability is more than just information exchange. It includes

systems, processes, procedures, organizations and missions over the life cycle and must

be balanced with information assurance. (DoDD 4630.5)

• Functional Interoperability: Also known as “syntactic interoperability” and “technical

interoperability”. It implies coordination across organizational lines. The higher the number of

stakeholders and their dependencies, the stronger the need to define common interface

specifications. All parties must agree to implement a common profile or interface template

(Sherif, 2009)

• Semantic Interoperability: focused on the consistent definition of data elements, ensuring

consistency from system to system and application to application. Achieved through the

development and use of standards for data and their definitions, minimizing interpretation errors.

(Sherif, 2009)

Source: (a) DoDD 4630.5 Interoperability and Supportability of Information Technology (IT) and National

Security Systems (NSS). (2007). Retrieved from http://www.dtic.mil/whs/directives/corres/pdf/463005p.pdf.

(b) Sherif, M. H. (2010). Handbook of Enterprise Integration. Boca Raton, Florida: Auerbach Publications; 1 edition

(November 9, 2009).

http://www.dtic.mil/whs/directives/corres/pdf/463005p.pdf

14

The Complexity of Interoperability

• Interoperability

– Procedures

– Equipment

• Hardware and software

– Standards

– Combination of the above

Source: USAF AFLCM/HNB (2013) Cursor On Target, COT-101.

Source Digital Interoperability – Optimizing the Kill Chain.

(2012) Precision Strike Technology Symposium.



15

Standards: Panacae or Pandora’s Box?

3 elements to VMF:

• Message Bearer(s) – Combat Net Radio protocols

• Message Header(s) – Required for message transfer

– Data not contained in

message

• Message Format(s)

– Catalogue of messages

CNR Protocols(Bearer Protocols)

Header(Message Header)

MessageCNR Protocols

(Bearer Protocols)

MIL-STD 188-220 MIL-STD 188-220MIL-STD 2045-47001 MIL-STD 6017

1st LastOrder of Transmission

VMF Data Stream

To be interoperable, systems must use

the same Message Bearer, Message

Header, and Message Format standards

Source: Robinson, T. (2011). International Data Link Society's Guide to Variable Message Format (VMF)

International Data Link Society (IDLS) (1.0 ed., pp. 2): IDLS.

188-220A188-220B188-220C188-220D

188-220D Ch1

TIDP-R2 R3 R4R5R6

R6+

60176017A6017B6017C

2045-470012045-47001A2045-47001B2045-47001C2045-47001D

2045-47001D Ch1

188-220A188-220B188-220C188-220D

188-220D Ch1

16

CAS Mission Performance

• Performance factors:

– Accuracy

– Flexibility

– Timeliness

– Situational Awareness


CAS Today: Non-standard, Non-Interoperable

DASC Direct Air Support Center

JSOTF Joint Special Operations Task Force

MTS Marine Tactical System

SADL Situational Awareness Data Link

JTAC ASOC Gateway

CoT

Link 16

(SATCOM) DASC

AFATDS JSOTF

F16B30

SADL

AFAPD

F16B40/50

AC130

AC130

A10C

B52 MTS

AV8B

VMF

F18A+/C/D/E/F

F16B40/50

F15E F18A+/C/D/E/F

A10C

AFAPD Air Force Applications Program Development

AFATDS Advanced Field Artillery Tactical Data System

ASOC Air Support Operations Center

CoT Cursor on Target

--------------- SADL

--------------- Link-16

--------------- CoT

--------------- AFAPD

--------------- VMF

---------------- MTS



DACAS CI Vision

JTAC ASOC Gateway

Link 16

(SATCOM)

JSOTF

F16B30

F16B40/50

CoT

SADL

F15E

AC130

AC130

F18A+/C/D/E/F

A10C

AV8B

F18A+/C/D/E/F

AH64

VMF

F35

A10C

B52

AH1

Airborne

Gateway

AC130

DASC

JFO

Ground Commander

JSOTF Joint Special Operations Task Force

SADL Situational Awareness Data Link

VMF Variable Message Format

ASOC Air Support Operations Center

CoT Cursor on Target

DASC Direct Air Support Center

Reduces number of legacy digital communication capabilities JTAC responsible for by 43% --------------- SADL

--------------- Link-16

--------------- CoT

--------------- VMF

19

Basic Research Question • Does an Integrated SE and T&E Process improve a SoS’s Performance?

Hypothesis • H1a Use of an Integrated SE & T&E Process contributes significantly and

positively to reducing SoS interoperability risk over time

• H1b Use of an Integrated SE & T&E Process has no significant impact on

SoS interoperability risk over time

• H1c Use of an Integrated SE & T&E Process contributes significantly and

negatively to SoS interoperability risk over time

Research Questions

20

End-to-End Process

Comparison Immediate CAS Mission Process

• Today, limited cross-Service

digital interoperability

– Reliant on voice

• The “To Be” Process

– Cross-Service interoperability

– Leverage digital M2M data

exchanges

The “As Is” versus the “To Be”

Source: JP 3-09.3 Close Air Support.

(2009). Chairman Joint Chiefs of Staff

Source: Robinson, T. (2011). International

Data Link Society’s guide to: Variable

Message Format.

21

Methodology

22

Methodology

• Utilizing an Integrated Systems Engineering and Test &

Evaluation Approach, conduct an analysis of the

Systems Engineering process for a “collaborative” SoS

• Validate the applicability of the approach for

“collaborative” SoS by

– Analysis of Risk Reduction and Test events results

• Improved Syntactic SoS interoperability

• Quantitative analysis of Semantic interoperability

– Analysis of “As Is” vs. “To Be” performance

23

• Data sources

– Interoperability Test and Risk Reduction Event Reports

• Population

– Constituent aircraft systems and ground kits (JTAC/JFO)

– Dynamic population due to asynchronous life cycles, availability

of resources at System , and nature of the collaborative SoS

• Validation of Research

– Provide quantitative methodology for semantic interoperability

Data and Validation

Examined SoS Models

1, SoS Needs & Objectives

5. SoS Enterprise

Strategic Plan

6. SoS Project Control

2. SoS Needs & Objectives

3. SoS Conops 4. SoS Scenario

2. Integrated SoS Acquisition

Strategy

1. Technology Investment & Development

Plan

3. Control System

Programs

2. SoS Capstone

Requirements Allocation

1. SoS Operational Architecture

1. Capability Requirements

2. SoS Systems Architecture.

4. Solution Design

9. SoS Analysis & Assessment

5. System Implementation

6. System Verification

7. System Capability

Development

10. SoS Verification

11. SoS Validation

3. SoS Operations

8. System Reengineering

and/or Disposal

Enterprise Activities

Development Activities

Operational Activities

Technical Activities

SoS Engineering Process Model (Source: Sage, Biemer 2007)

Sources: Dahmann, J., Rebovich, G., Lane, J., Lowry, R., & Baldwin, K. (2012). An implemented view

of systems engineering for systems of systems. Aerospace and Electronic Systems Magazine, IEEE,

27(5), 11-16. doi: 10.1109/MAES.2012.6226689

Sage, A. P., & Biemer, S. M. (2007). Processes for System Family Architecting, Design, and

Integration. Systems Journal, IEEE, 1(1), 5-16. doi: 10.1109/JSYST.2007.900240

DoD SoS Wave Model SoS Engineering Process

Quantifying Semantic Interoperability • Matrix tables limited to qualitative depiction

of interoperability

• Non-compliance impacts with data field, display and special considerations requirements

• “Mandatory” with “Display”

• “Mandatory”, no display with “Special considerations”

• “Discretionary with “Display”

Bearer Protocols Match?

Are Required Protocols

Matching?(Syntatic)

Message Header Match?

Message Standard Match?

Start

Not Interoperable

Examine Data Field Element Minimum

Implementation

Are There Discretionary

Data Authentication

Code (DAC) Category (CAT)

Fields?

Are There Mandatory Data Authentication


Fields?

Examine Data Authentication

Code (DAC) Column

No Impact

Are Mandatory Display Column

Fields Implemented?

No

Yes

No

Yes

No

Interoperability Broken

Are Discretionary

Data Authentication


Fields implemented?

NoNo

Interoperability Broken

Examine Display

Column Fields

No

Yes

Yes

Are There Display Field

Requirements?

Examine Special

Consderations

NoAre There Special

ConsiderationsNo Impact

No

Are Display Requirements Implemented?

Yes

Method & Location System

SpecificTabular Display

Graphical Display

No

Interoperability Maintained

Yes

Display Not Prohibited

Interoperability Impacted

Are Special Considerations Implemented?

No

Yes



Interoperability Impacted

System A System B System C System D System E

System F

System G

System H Tx/R

x

Message 5 Rx/Tx

26

Expected Outcomes

• Improved Syntactic Interoperability between SoS

constituent systems

• Improved Semantic interoperability between SoS

constituent systems

• Improved SoS Mission performance (timeliness)

27

Summary

• Challenges to SoSE are unique

– Disparate “asynchronous” lifecycles

• “No single approach can solve these emerging

problems, and thus no strategy is best for all projects” (Gorod, Sauser, and Boardman, 2008)

• An Integrated Systems Engineering Development and

Test & Evaluation Approach is valid for a “collaborative”

SoS

– SoS approach must ensure synchronization of upgrades

28

References • (a) Dahmann, J. (2012, 19-22 March 2012). Integrating systems engineering and test & evaluation

in system of systems development. Paper presented at the Systems Conference (SysCon), 2012

IEEE International.

• (b) Sage, A. P., & Biemer, S. M. (2007). Processes for System Family Architecting, Design, and

Integration. Systems Journal, IEEE, 1(1), 5-16. doi: 10.1109/JSYST.2007.900240

• (c) Choi, D., & Sage, A. P. (2012). A framework for interoperability assessments in Systems of

Systems and Families of Systems. Information, Knowledge, Systems Management, 11(3), 275-

295.

• (d) Maier, M. W. (1998). Architecting principles for systems-of-systems. Systems Engineering,

1(4), 267-284 .

• (e) Chen, P., & Clothier, J. (2003). Advancing systems engineering for systems‐of‐systems

challenges. Systems Engineering, 6(3), 170-183.

29

References

• (f) Keating, C., Rogers, R., Unal, R., Dryer, D., Sousa-Poza, A., Safford, R., .Rabadi, G. (2008).

System of systems engineering. Engineering Management Review, IEEE, 36(4), 62-62.

• (g) Gorod, A., Sauser, B., & Boardman, J. (2008). System-of-systems engineering management: a

review of modern history and a path forward. Systems Journal, IEEE, 2(4), 484-499

• (h) Dahmann, J., Rebovich, G., Lane, J., Lowry, R., & Baldwin, K. (2012). An implemented view of

systems engineering for systems of systems. Aerospace and Electronic Systems Magazine, IEEE,

27(5).

• (i) Joint Publication (JP) 3-09.3, Close Air Support. (2009). Chairman Joint Chiefs of Staff

• (j) Sherif, M. H. (2010). Handbook of Enterprise Integration. Boca Raton, Florida: Auerbach

Publications; 1 edition (November 9, 2009).

• (k) Hura, M., McLeod, G., Larson, E., Schneider, J., & Gonzales, D. (2000). Interoperability: A

Continuing Challenge in Coalition Air Operations Santa Monica, California: Rand Corporation.

30

Back-Up Slides

31

The CAS Challenge • http://www.youtube.com/results?search_query=500lb%20bomb%20dropped%20

on%20us%20soldiers%20by%20mistake&sm=1 – Watch timeframe 0:00-0:25

– Warning: Explicit language

• http://www.youtube.com/watch?v=wWrgUZbuhiw

http://www.youtube.com/watch?v=wWrgUZbuhiw










Documents

An Analysis of An Integrated Systems Engineering & Test