Adaptable C5ISR

Instrumentation

Mission Command and Network Test Directorate

Prepared by Mr. Mark Pauls

U.S. Army Electronic Proving Ground (USAEPG)

21 May 2014

U.S. Army Electronic Proving Ground Advanced Electronics & Network Systems

Mission Command & Network Test Directorate

US Army Electronic Proving Ground

2000 Arizona Street

Fort Huachuca AZ 85613

UNCLASSIFIED

Distributed T&E Instrumentation

• C5ISR systems have evolved into very rapid Commercial-off-the-shelf (COTS) technology procurements

• Increasing use of these new and rapidly evolving technologies presents a challenge to the Test and Evaluation (T&E) community

• Shrinking procurement cycle forces the T&E cycle to also decrease

• To adequately verify these systems a robust instrumentation package must be developed and implemented to proved stimulation/simulation, data collection, reduction, and analysis

• Non-platform specific instrumentation suite of software tools that are embedded and LVC

• C5ISR system instrumentation modifications and innovative without the lengthy procurement process

• Instrumentation platform must support a wide array of standard and custom input/output (I/O) functions.

• Software tools must be reprogrammable to interpret the data streams for C5ISR system under test.

• Meet constraints of size, weight, and power (SWaP)

Challenge Solution

2

UNCLASSIFIED

Shrinking Test Technology

Resources

• Constrained Budgets

• Shrinking Number of Test Events/Increased Complexity

• Test Completion & Successful Metrics

• DOE Driven events to gain the most possible from a single event

• Aging T&E Work-force

• Limited Academia T&E Professional Programs

3

UNCLASSIFIED

Shift Left Embed Instrumentation Early

• Estimate 70% of the defects could be

discovered pre-milestone “B”

• Find defects early and avoid the

unnecessary schedule and cost over runs

• Left shift in testing takes care of the

ailments as they occur and ensures

minimal after effects.

Shift - Left

Current T&E Cycle

Improve Acquisition Outcomes

Dr. C. David Brown “T&E in Defense Acquisition” and Dr. Steve Hutcheson “Shift Left”

4

UNCLASSIFIED

Problem Definition

• Test and Evaluation Legacy

• Solutions are expensive

• Do not support the Cradle to Grave

• Do not standardized performance measurement

• Do not support lifecycle of Systems Testing

• Hardware-based solutions

• Do not leverage Systems Computing Resources

• Customized to supporting a single system test

• Increase risk with the employment of new complex systems

• Cumbersome and require maintenance

5

UNCLASSIFIED

Operational Instrumentation View

Spectrum Monitoring Sim/Stim Data Collection Data

Reduction/Analysis

6

UNCLASSIFIED

The Big Picture

Integrated Data Reduction and Analysis

•Multiple Sources of Data

• Embedded Agents

• VM Collectors

•Seamless Instrumentation Infrastructure

• Test Control

• Common DB

• Meta Data

•Data collected enterprise-wide must be fused for comprehensive analysis

Testing Advantages

•Legacy – Single unique test instrumentation

•Golden Eye Solution – Light weight hardware with installed software

•Embedded Solutions – Software embedded per milestone “B” – Life Cycle Testing – Improved Acquisition Systems

7

UNCLASSIFIED

Instrumentation Approach

Simulation

Collection Analysis

Visualization

8

UNCLASSIFIED

Solution

What is “embedded” instrumentation?

Why should we embed and virtualize

instrumentation?

9

UNCLASSIFIED

Next Generation Tools

• Machine Learning Embedded Test Tools - Software Development Kits (SDKs)

• Use of SDK’s to develop collection and analysis threads

• Use of SDK’s to validate networks

• Use of SDK’s to ensure compliance

• Use of SDK’s to support a wide array of data standard

10

UNCLASSIFIED

Application Programming Interface (API)

• Development of standard API’s will ease the work of programming graphical user components

• Over time the developed API’s would form a library that includes specifications for routines, data, structures, object classes and many other variables

• API technology enables programmers to build hybrid analytics solutions

• The use of functional API has the potential to transform unstructured data into intelligence

11

UNCLASSIFIED

Machine Learning API’s

• Selection of the “Best Model”

• Statistical Based Models

• Biased based on model used to develop the API

• Limits of the model curve not infinite

• Human Genome Models

• Un-biased based on repetition of occurrence

• A lot like the human brain learns as things occur

12

UNCLASSIFIED

Embedded Stimulation

Virtual Environment

Virtual

Switch

SUT Embedded

Stimulator

Virtual Machine

SUT Embedded

Stimulator

Virtual Machine

• Single server platform • Multiple embedded

stimulators

• Multiple virtual environments

• Stimulate message/network traffic

• Between different virtual systems

P

H

Y

S

I

C

A

L

N

E

T

W

O

R

K

Instrumentation

IMS – Network Traffic

Stimulation

SUT

Embedded

Stimulator

SUT

Embedded

Stimulator

Controlled Environment

13

UNCLASSIFIED

Network Effects Emulation System

•Network Effects • Latency, Packet loss, Packet corruption,

Packet duplication, Packet reordering, Bandwidth limitation

• NE2S provides a Cyber testing capability by simulating a wide range of computer and network-based effects that can be centrally managed and controlled during a training and/or test event

• NE2S is a TRMC program managed by the Joint Coalition Warfare Office/J7. The program is at the TRL 6 maturity level

•Host Effects • Blue Screen, Erratic Mouse, Keyboard

Lock, Forced Reboot, Sound Damage, Monitor Disruptions, CPU Effects, Memory Effects

Tester

Distributed

Network

Access

Switch

`Mission Command

Platform

`Mission Command

Platform

Host Based

Effects

Generator

(Software)

Web browser

.HT

TP

S.

SSH with public

key authentication

CPU Effects

Mem Effects

Net Effects

NE2S Master Station

Tester

Starship / StarGEN

.ST

AX

.

14

UNCLASSIFIED

EPG

Fort Dix established network

Redstone Test Center

Joint Staff J6

(C4AD)

EPG-JBLM

1: Cyber & IA Test Facility

2: Realistic Battlefield Emitters

3: Test Events: NetWarrior, Manpack, Prophet

4: Large Range, controlled airspace, controlled

RF spectrum

5: Jamming & other EW.

Fort. Hood Operational Test Command

& Central Technical Support Facility

PEO-C3T & DCGS

JITC

Distributed Test Capability

15

UNCLASSIFIED

Leveraging Academia Partnerships

for Research Concepts

Network Topology / QoS FEATURE

EXTRACTIONFEATURE

SELECTIONDATA

MININGPREPROCESSING

INTERPRETATION& EVALUATION

Log -> Flow Transformed data Reduced data Patterns Knowledge

DATA ACQUISITION

Network packet trace (log)

Data Mining

• Characterize network behavior

via statistical analysis

• Detect significant

changes/events by comparing

adjacent time windows

• Identify root cause for changes

16

UNCLASSIFIED

• Identifying S&T Gaps for Next Generation Embedded Test Tools

• Verification and Validation of Embedded Test Tools

• Adapt Policy to allow Co-Use of DT/OT Embedded Tools for Life Cycle Testing

• Testing Community/PM’s agreement to embedded test tools for life cycle testing

• Embed Cross-Domain solutions

Challenges

17

Questions?

Electronic Proving Ground

Phone: (520)538-8043

Mr. Mark Pauls

mark.k.pauls.civ@mail.mil

18