1

Cyber T&E Standards Panel

• Why Cyber T&E Standards?Mr. George Wauer, Touchstone POCs, LLc

• Test and Training Enabling Architecture (TENA)Mr. Gene Hudgins, TRMC

• Cyber Range Environment VV&AMr. Ryan Kelly, JHU/APL

• DECRE Cyber Range Interface SpecificationMr. David Gerrek, J7-JIOR

• Cyber Range User’s GuideCol Burton Catledge, USD(AT&L/C3CB)

2

Why Cyber T&E Standards?(Concept Discussion)

George WauerTouchstone POCs, LLc

17 March 2016

• Problem space• Cyber T&E Infrastructure (CT&EI)• DoD Enterprise requires a Federated approach• Parsing a T&E event by responsibilities• OIPT (like) and Working Groups• Enterprise-wide Architecting and Systems Engineering (EASE)

3

Cyber Test Ranges Areas of Concern

• Capacity: – Systems within the acquisition process (100s of programs) – Legacy systems being assessed for cyber issues (NDAA 16 requires). (test-fix-test cycles)– RMD for FY16 added some capacity (doubt its solved)

• Capability: – Ability to establish operationally representative test environments (DCO & OCO) lacking– VV&A is at best ad hoc– Red Teams and Blue Teams are not consistent even when available– T&E processes too slow for need

• Cohesion: – Several material development commands (or PMs) establishing individualized approaches

and investments to solve native needs– Builds “stovepipe” solutions

• Unique solutions to common problems - done in isolation• Fragmented approach produces unwarranted duplication, and partial solutions which

may or may not integrate into a DoD-wide enterprise test capability – An integrated Cyber T&E Infrastructure across the C/S/As required to test the way we fight

(Notional)

4

Systems are Required to be “Effective” in aCyber Contested Environment

Events FidelityComplexityEvaluationDemand

Cyber Ranges (30)Almost no mission

effectiveness

Existing T&E Infrastructure (100s of sites)Product and mission effectiveness(non cyber)

TestSupply

(Cyber capable)

• Stand alone• SoS (10s of systems) • Low - Operational

• S&T – OT&E• 3000+/yr

Expand to handle:# events ComplexityFidelity Mission Effectiveness

($$$)

Cyber T&EInfrastructure

or

(Notional)

5

Army HWILs

Navy HWILs

AF HWILs

DISA HWILs

DISA SILs

Army SILs

NavySILs

AFSILs

AFRanges

ArmyRanges

NavyRanges

AFISTFsNavy

ISTFs

Hardware-in-the-Loop

Laboratories (HWILs)

System

Integration

Laboratories

(SILs)

Installed System

Test Facilities

(ISTFs)Open-Air Ranges

Realistic Mission

Environments

JMETCJMN/JIOR

RSDPsPSDPs

Parts of TSMO

AF 346th

USS Secure

AF 46th

Det #2

Cyber specific

T&E Capabilities

such as:

NCR

Cyber Test and Evaluation Infrastructure (CT&EI)

External Partners: Industry,

Non-DoD, Allies, Academia, etc

Common Architecture

and Standards with Training Community

Distributed Access to Readily Available

Cyber T&E Capabilities

(Notional)

6

• The Cyberspace T&E Infrastructure (CT&EI) is defined as: a set of capabilities (information technology [IT] infrastructure, instrumentation, tools, processes, facilities, and workforce) which can be integrated to generate an environment that operates at the appropriate classification levels and controls to provide a representation of cyberspace to support T&E events or functions.

• The CT&EI will be multi-purposed to conduct a broad range of science and technology, research and development, T&E, and when called upon, capable of augmenting training activities.

(Notional)

7

CoordinationDirector, TRMCEA Cyber

Test and IT Ranges

USD(AT&L)

Focal Point/CCT(Investments)

CIMB

Army EA Cyber

Training and IT Ranges

USD(P&R)

Biennial ReportAdvisory Board

Cyber Test and Evaluation

Infrastructure(CT&EI) -

Includes MRTFB like sites

Federated Management

Cyber Training Ranges

(school houses, etc.)

Possible EA Governance Structure

coordination

(Notional)

8

Federated Management*Levels

Management OptionsCyber T&E Infrastructure

Cognizance Collective

Coordinated

Controlled(Own)Concert

EfficientEnterprise

Field Activities

In Federated Management each level has:• Different authorities with “Tiered Accountability”• Requires different communications with C/S/As (visibility)• Governance (Authority, Direction, Monitor, Remediation)

* Based upon 36th Annual International Conference on Systems Sciences - 2003

(Notional)

9

Governance Framework(Federated Management)

* Authority must match Responsibilities• Leverages Processes at Component Level to Get the Majority of the Job Done• Components Accountable for Execution

Authority

Direction/Guidance Monitor

Affirmation/Remediation

Implementation Monitor

Component Plan

Type I, II, III

Component(Execution)

Top Level Agent*Issue paper as needed

Affirmation/Remediation

Tiered

Accountability

(Notional)

10

As an Example consider: Cyber Event

Architecture Components

Resources

Traffic

Generation

Defensive

Packages

Asset

Management

Exercise/Test

Scheduler

Health &

Status

Range

Sanitization Environment

Range Setup, Control, Operation

Services

Supports: System Development

Models

Army Prototypes

National & DoD Agencies

CapabilitiesAir Force Navy

…

WebTransportationMaritimeSatelliteAirborneRadars

Adapter Adapter Adapter Adapter Adapter Adapter

Open Standards

Network

Network

ExercisesConcepts

SCADA

Range

Automation Threat

PackagesRange

Validation

Distributed

Operations

Command &

Control

Visualization

Technology Development

Targetable

Data Collection

Metrics

Core Services

Data Archival

Instrumentation

Data Analytics

Radars Ships/Missiles

Accessible, Integrated

Environment

Emulation

Mission Traffic

Emulation

Telecom

Adapter

Blue Red Gray

COCOM Activities

Range

(Notional)

11

What needs to have standards?

• Tailored to match Federated Management Level– Recognize responsibilities of the respective Components– Develop an Architecture Vision Doc. (include: ConOps, attributes, etc)

– Work with Components to establish Standards across the CT&EI

• OIPT construct w/Working Groups (potential)– Technical– Security– Environments (incl. VV&A processes/procedures)– Operations– Node Credentialing?– Scheduling and long range planning– others

12

2018

Capability

Increments

4

T

H

E

S

T

A

T

E

A

I

R

F

O

R

C

E

N

A

V

Y

A

R

M

Y

Collaborative EASEEnvironment

EASE Oversight

Revised

Standards/SLAs

(e.g., Comms

Requirements)

Enterprise-wide Architecting & Systems Engineering (EASE)

The architecture and

SE processes must

be collaborative

FunctionalDirection(e.g., CT&EICapstone)

SE Guidance

AV, OV, SV

Collaborative

EASE Environment

Area of Detail (Example)

Temporal Direction(e.g., CT&EIIncrement #4)

Enterprise

4th Estate

PMs

Archand SE

Air Force

PM

Archand SE

Navy

PM

Archand SE

ArchArchand SEArmy

PM

Archand SE

Archand SE

COI Req’ts

Archand SE

CT&EIComponent Area

FunctionalDirection

SE Guidance

AV, OV, SV

SE Guidance

CAIV

$$$

Feedback

2024

2022

2020

Community (S&T, R&D, T&E, etc) Requirements

CT&EI architecturalvision

CT&EI Capstone

13

Back ups

13

14

TemporalType III

TechnicalType II

Standards

Data Structure

Safety

Etc.

Timing for RequiredCapability

(Increments?)

Direction and Guidance

Type I

DoDDs, DoDIs and Guides

(Roles andResponsibilities)

(Processes andProcedures)

PolicyType I

15

Federated Management Levels(Simplified Description)

• Collected: Minimal or no coordination among members required for the individuals to be productive. Group productivity is the aggregate of the individual efforts.

• Coordinated: Individual efforts, but success depends on ability to coordinate efforts. Managing interdependencies, should have ordered and progressive integration of tasks and within a process structure (group calendar, routine project tracking, etc.)

• Concerted: Individual efforts, but performance of any one influences others’ ability to perform. Everyone on the same sheet of music and lead by a “conductor” for emphasis, timing, etc. Lots of communication throughout the federated structure.

16

• Tiered Accountability.* A federated management approach to achieve an effective and efficient outcome, where multiple levels of organizations exist, each having respective authorities. The DoD can be defined as a set of tiers, which, as a minimum, includes Enterprise, Capability, and Component levels. Each tier of the DoD governs the areas for which it is responsible and maintains consistency with guidance from higher tiers.

*DoDI 8330.1

1717

1818

1919

2020

2121

2222

2323

2424

2525

2626

2727

2828

2929

3030