Test and Evaluation of Cyber Systemsitea.org/images/pdf/conferences/2015_Symposium... · networks • Hackers may be multiple motivations, including profit, protest, or because of

Test and Evaluation

of Cyber Systems

Aug 18, 2015

Arlington, VA

G.A. (Fred) Wright, PhD

404.407.7296

Cell: 404.840.7652

[email protected]

Contact Information

mailto:[email protected]

• Build intuition related to cyber security

technology, risks, and methodologies

• Investigate systems approaches, threat, risk

evaluation, and countermeasures

• Consider challenges and approaches of test and

evaluation (T&E) of cyber systems

Objectives and Themes

• Cyberspace and Cyber Systems

• Threats

• Definitions

• Business / Mission Assurance

• Information Technologies

• T&E Challenges

• Metrics and measures

• Planning

• Configuration / test execution

• Data Reduction and Analysis

• Technologies for Cyber Testing

Outline

Introduction to Cyber Systems

• Introduction to Cyberspace,

• Understanding the threat

• Cyber security definitions

• Business enterprise view

Cyber is such a perfect prefix. Because nobody

has any idea what it means, it can be grafted

onto any old word to make it seem new, cool --

and therefore strange, spooky.

New Yorker Magazine, Dec. 23, 1996

Reference: Wikipedia - Information Age - A Visualization of the various routes through a portion of the Internet.

All I knew about the word "cyberspace" when I

coined it, was that it seemed like an effective

buzzword. It seemed evocative and essentially

meaningless. It was suggestive of something,

but had no real semantic meaning, even for me,

as I saw it emerge on the page.

William Gibson

Current State, Unattributed Quotes

• “The state of cyber security today is a complete failure…If you haven’t been hacked you have nothing of interest to steal”

• “fundamental trust models in cyberspace are broken; there is no technology out there today that reflects trust; 100 years from now we will realize we were in a lawless state”

• “why do we lack systems understanding, holistic design principles, risk management, and training in our enterprise systems?”

• “we are our worst enemies…the problem is too huge…we cannot conceptualize it, cannot worry about it”

• “it’s going to take a ‘BP oil spill of data’ event to wake us up”

Agenda

• Introduction to Cyberspace


• Security definitions

• Business Enterprise View

Current State is Rapidly Evolving & Expanding

• Hacker (1960’s)

• A person who enjoys exploring the details of programmable systems and stretching their capabilities

• “WarGames” (1983)

• A young hacker starts the countdown to World War 3.

• Computer Viruses (1980’s)

• Tool era - Self-replication & connectivity

• Hacktivism (1990’s)

• WANK Worm … to Anonymous & Lulz (2011)

• Cyber Criminals (2000’s)

• Financial theft, illicit trade

• Cyber Espionage (last decade)

• Characterized by persistence

• Cyber Kinetic Attacks (emerging)

• Primarily nation-state based, target physical systems

RQ-170 “Capture”

Current State is Rapidly Evolving

• Remarkable change in attack motivation from our IT Systems to our Enterprises

• Around 2005, saw attacks shift from individual IT systems to commercial enterprises

• Unprecedented transfer of wealth, not just IP but also enterprise strategies

• Organized crime and nation-state involvement

• Key threat shift: preparation and patience

• Not typical hacking – normal IT tradecraft used, but the technology is mainstream

• Espionage: reconnaissance, exfiltration, exploitation, profit

• New paradigms – “we have no idea what’s out there”

Hacking

• In computer security and everyday language, a hacker is someone who breaks into computers and computer networks

• Hackers may be multiple motivations, including profit, protest, or because of the challenge

• The subculture that has evolved around hackers is often referred to as the computer underground but it is now a somewhat open community

• Hacking is not necessarily bad

Reconn.

Scanning

Gaining Access

Maintaining Access

Covering Tracks

Generic Buffer Overflow “Exploit”

Buffer

Instruction Pointer

Malicious Code

• Target Credit Card

Info Incident

Recent Case

Target Point-of-Sale (PoS) breach:What do we know now?

Dell SecureWorks:“Inside a Targeted Point-of-SaleData Breach”

• Threat indicators reported twice by sensors, several weeks apart, before exfiltration began

• Began with Spear Phishing attack on Target’s HVAC provider

• Multi-step process to gain access to PoS network

• Multi-step process to aggregate and exfil data

• Limited information on how threat moved laterally within the network to PoS network

Agenda



• Cyber Security definitions

• Business enterprise view

What’s a Cyber System?

• Computer + Software + Internet = Cyber System?

• Local/Private Networks?

• Mobile/wireless- GSM, 3G, 4G?

• Combinations of these?

From US Air Force Brief on Cyberspace

Cyberspace/security Regimes

1: Physical

2: Data Link

3: Network

4: Transport

5: Session

6: Presentation

7: Application

Open Systems Interconnect

(OSI) Reference Model

Physical Network

Logical Network

Social/User Network

Multiple disciplines in

a complex system of

systems

8: User

Typical Cyber Attacks: Upper layers

1: Physical

2: Data Link

3: Network

4: Transport

5: Session

6: Presentation

7: Application

Cyber attacks occur at all layers; but attacks (on the internet)are prevalent at the application layer

Typical Cyber Attacks: Upper layers

1: Physical

2: Data Link

3: Network

4: Transport

5: Session

6: Presentation

7: Application

Many cyber attacks enter through the application layer with a goal of controlling computers, collectingdata, or inserting data

Typical Electronic Attack (EA): Lower layers

1: Physical

2: Data Link

3: Network

4: Transport

5: Session

6: Presentation

7: Application

Electronic attacks enter through the physical layer with a goal of disrupting or deceiving

Typical Electronic Attack (EA): Lower Layers

1: Physical

2: Data Link

3: Network

4: Transport

5: Session

6: Presentation

7: Application

Electronic Attack

Battlespace Components

C2 Center

Air Defenses

Cyber System

Any device or system participating in a local or

global network of interdependent information

technology infrastructures, telecommunications

networks, and computer processing systems

What is Cyber Security?

Computer security - protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to remain accessible and productive to its intended users.

Reference: http://en.wikipedia.org/wiki/Computer_security, http://en.wikipedia.org/wiki/Information_security, http://en.wikipedia.org/wiki/Network_security, http://www.merriam-webster.com/dictionary/cybersecurity

Information security - protecting information and information systems from unauthorized access, use, disclosure, disruption, modification, perusal, inspection, recording or destruction.

Cyber security - measures taken to protect a computer or computer system (as on the Internet) against unauthorized access or attack.

Network security - consists of the provisions and policies adopted by the network administrator to prevent and monitor unauthorized access, misuse, modification, or denial of the computer network and network-accessible resources

Information Assurance (IA)

• Measures taken to protect and defend sensitive

information from an adversaries efforts to deny, destroy,

degrade or disrupt information or information systems.

• Measures taken to ensure that information is available,

reliable, defendable, and verifiable.

• Measures taken to ensure that information and

information systems implement requisite protection,

detection, and reaction capabilities.

IA Model is Risk and Threat-Based

Common Criteria for Information Technology Security Evaluationhttp://www.commoncriteriaportal.org/

Joint Pub 3-12

Cyberspace operations: The employment of cyberspace capabilities where the primary purpose is to achieve objectives in or through cyberspace

Cyberspace Operation

• Information Assurance – Making systems defendable

• Cyberspace Operations

• Enterprise/Network Operations (NetOps) – Running and managing the systems

• DoD Global Information Grid Operations (DGO)

• Defensive Cyberspace Operations (DCO) – Monitoring and responding to incidents (e.g., attacks, intrusions…)

• Cyberspace ISR – Discovering information (gathering intelligence)

• Offensive Cyber Operations (OCO) – Attacking systems

Cyberspace Operations Definitions

ISR in Cyberspace: Impacts of Attacks/Defense?

• Denial

• Disruption

• Degradation

• Destruction

• Deception

Issues with Information and Operations

• Cyberspace Attack: Cyberspace actions that create various direct denial effects in

cyberspace (i.e., degradation, disruption, or destruction) and manipulation that leads to denial

that is hidden or that manifests in the physical domains. These specific actions are:

• (a) Deny. To degrade, disrupt, or destroy access to, operation of; or availability of a target by

a specified level for a specified time. Denial prevents adversary use of resources.

• l. Degrade. To deny access (a function of amount) to, or operation of, a target to a level represented as

a percentage of capacity. Level of degradation must be specified. If a specific time is required, it can

be specified.

• 2. Disrupt. To completely but temporarily deny (a function of time) access to, or operation of, a target

for a period represented as a function of time. A desired start and stop time are normally specified.

Disruption can be considered a special case of degradation where the degradation level selected is

100 percent.

• 3. Destroy. To permanently, completely, and irreparably deny (time and amount are both maximized)

access to, or operation of, a target.

• (b) Manipulate. To control or change information, information systems, and for networks in a

manner that supports the commander’s objectives, including deception, decoying,

conditioning, spoofing, falsification, etc. Manipulation uses an adversary’s information

resources for friendly purposes.

From JP 3-12

• Computer Network Attack (CNA): Includes actions taken via computer networks to disrupt, deny, degrade, or destroy the information within computers and computer networks and/or the computers/networks themselves.

• Computer Network Defense (CND): Includes actions taken via computer networks to protect, monitor, analyze, detect, and respond to network attacks, intrusions, disruptions, or other unauthorized actions that would compromise or cripple defense information systems and networks.

• Computer Network Exploitation (CNE): Includes enabling actions and intelligence collection via computer networks that exploit data gathered from target or enemy information systems or networks.

Previous Terms (still widely used)

What is Cyber Warfare?

Network Centric Warfare is not Cyber Warfare

Network Centric Warfare is using cyber technology (computers/networks) to improve performance in Land/Sea/Air/Space domains

Tightening the (Observe, orient, decide, act (OODA) loop)

Electronic Warfare (EW) is not Cyber Warfare BUT… there is significant overlap

EW can have provide cyber warfare effects

And vice versa

In Cyber Warfare, the targets are in Cyberspace!

Whether defended targets or adversary targets

Cyber warfare might be considered a subset of Network centric warfare

When Information Becomes Digital Data

Concerned with:

• Data Access

• Data Structure

• Data NetworkSensitive

Data

ControlledData

PersonnelData

OperationalData

Confidentiality Integrity

Availability

Other Networks

PacketSwitch

Gateway

FileServer

Bridge

C-I-A Concerns: Access to the Data

• Confidentiality

• No disclosure

• Only those who need to see data should see it

• Integrity

• No alteration

• Only those allowed to alter data can modify it

• Availability

• No interruption

• Everyone who needs to access data can access it

Confidentiality Integrity

Availability

Cyberspace Visualization

Science concerned with the presentation of data

Understanding and extraction of information from data

Which techniques work best for Cyberspace?

Agenda



• Security definitions

• Business Enterprise View

• Objectives of IT enterprise is to support

“business” strategy and processes

• Business processes often utilize numerous IT

components

Business and Enterprise View

What processes and functions?

Image: http://jeffsutherland.org/oopsla97/hung.html

Co

nti

nu

um

of

Net

op

s Fu

nct

ion

s

Enterprise Management(Business Process or Mission Assurance)

EnterpriseAnd Network Management(Service Management)

Network/DeviceManagement

• Business-centric

• Service-centric

• Application-centric

• Infrastructure-centric

Incr

easi

ng

Mat

uri

ty o

f P

rod

uct

s &

Pro

cess

es

Trade Space: Progression and Complexity

• 26 processes for devising and managing IT services

• Focus on providing services levels (service level agreements (SLAs))

• Primarily addresses network infrastructure and telecom but SW management processes added

• Five parts, includes security processes in each:

• 1. Service Strategy

• 2. Service Design

• 3. Service Transition

• 4. Service Operation

• 5. Continual Service Improvement

Example Systems Engineering Process: IT Infrastructure Library (ITIL)

DoD Cyber Commands

USSTRATCOM

USCYBERCOM

24th AF 10th Fleet ARCYBER MARCYBER

AF Space

Command

Regional

COCOMs

Defense Information

Systems Agency

National Security

Agency

• IT services provide infrastructure for business processes

• Business performance metrics provide a basis for assessing security issues and incidents

Business management, IT enterprise management, and security must come together

Business Process and Metrics Provide Context


• Threats

• Definitions



• T&E Challenges


• Planning




Outline

Overarching Test Issues forNetwork-centric Systems

• Lack of operators

• Large number of nodes/Equipment

• Complex scenarios

• Distributed systems

• Variety of information exchanges

• Measuring effectiveness

Constructing the test environment!

Emerging/Evolving Complexity in Testing C2/C4I Systems

• Establishing and maintaining C2 system of systems and T&E capability• Numerous interfaces, message types, networks, comms,

and applications

• Distributed environment - geographic separation

• Multiple systems in joint environment – One system under test (SUT)

• Cost and bandwidth required to establish and maintain distributed M&S

• Software maintenance with changing C2 systems

System(s) Under Test (SUT)

Communications

(Transmissions) Systems

Applications (Information

Systems)

Workstations or Terminals

Sensors

Doctrine

System of Systems

Digital Networking

Equipment (Servers/

Gateways/Bridges/Routers…)

C2 Facilities (Buildings,

Vehicles, Enclosures…)

• Complex systems

• Users / operators – variable vulnerabilities

• Voluminous heterogeneous data

• Vulnerabilities are often difficult to predict/find

• Threat agents and vectors are not easy to characterize

• Metrics for some cyber operations

• mapping to mission effectiveness

• Offense / defense as symbiotic pair

Challenges with Cyber Systems

Refine

T&E as Part of Single-StepSystem Development Process

ConceptDefinition ?

Refine

Stop Development

SubsystemDevelopment ?

Refine

Stop Development

PrototypeDevelopment ?

Stop Development

T&E

T&E

T&E

ProductionSystem ?

Refine

Stop Development

T&E Release

Continue

Continue

Continue

Need

RISKACCEPT-

ABLE?

NO

DETERMINEOBJECTIVES

PRE-TESTANALYSIS

TEST EVALUATE

PRODUCT:KNOWN

RISKSOLUTION

IMPROVE

YES

T&E Phases

Pre-testTest

Event

Execution

Post-Test

• Risk Management Framework

• Artifacts should be helpful in T&E

• OSD-OT&E Memo, Aug 1, 2015: T&E Requirements

• Cooperative Vulnerability and Penetration Testing

• Adversarial Assessments

• Cybersecurity Test and Evaluation Guidebook, July 1, 2015

• Understand Cybersecurity Requirements

• Characterize the Cyber-Attack Surface

• Cooperative Vulnerability Identification

• Adversarial Cybersecurity DT&E

• Cooperative Vulnerability and Penetration Assessment

• Adversarial Assessment

DoD Guidance

• The PM will take full advantage of DoD ranges, labs, and other resources.

• DT&E activities will start when requirements are being developed to ensure that key technical

requirements are measurable, testable, and achievable.

• The DT&E program will support cybersecurity assessments and authorization.

• The PM will develop a strategy and budget resources for cybersecurity testing. The test program will

include, as much as possible, activities to test and evaluate a system in a mission environment with

a representative cyber threat capability (additional guidance is included in the DAG).

• For Major Defense Acquisition Programs, the DT&E T&E Master Plan (TEMP) approval authority will

provide the Milestone Decision Authority (MDA) with an assessment at each milestone review or

decision point.

• Beginning at Milestone (MS) A, the TEMP will document a strategy and define resources for

cybersecurity T&E.

• Beginning at MS B, appropriate measures will be included in the TEMP and used to evaluation

operational capability to protect, detect, react, and restore to sustain continuity of operation.

DoDI 5000.02, January 7, 2015

Cyber T&E Phases


• Threats

• Definitions



• T&E Challenges


• Planning



Outline

Measures and Metrics

ISSUES

MOE

1

MOE

2

MOE

n

MOP

1.1

MOP

1.2

MOP

1.n

.

.

.

.

.

.

TESTPARAMETER1.1.1

TESTPARAMETER1.1.2

TESTPARAMETER1.1.m

.

.

.

Pre-Test: From Issues

to ObservablesMOE = Measure of Effectiveness

MOP = Measure of Performance

• What questions do we need to answer?• How much testing?• How to test?• How to tailor test for life cycle?

. . .

. . .

. . .

. . .

ConductPlanning forOperation

Picture/Awareness

Target NominationProcess

Number/Types ofTracks

Interoperability ofLink feed interfaces

Timeliness/Accuracy ofTrack Updates

Commonality/Relaventnessof Awareness

Effectiveness ofSupportedCollab. Protocols

ProcessExecutionTime

Critical OperationalIssues

Measures ofPerformance

TESTING MEASURES

PERFORMANCE

SystemLevel

Mission

Does System provideFor effectiveMission execution

Measures of Effectiveness

Force levelTasks

Metrics Breakdown

…

MOP/Technical Performance Breakdown

Number/Type ofTracks

Interoperability ofLink feed interfaces

Timeliness/accuracy ofTrack Updates Common

Awareness

Effectiveness ofCollab. Tasks

ProcessExecutionTime

Measures of Performance

Link Message LatencyTask Latency

AlternativesAnalyzed

Number ofTargets in Plan

Technical PerformanceParameters

GeographicalDifferencesIn COP…

…

Effective Bandwidth … Quality of Service

Interactive/ExchangeAOC – AADC –JFACC

… …SystemPerformanceParameters

Basic Network Centric T&E Process

Define Objectives, Measures, and

Data Requirements

Define Test Network

Configuration

Generate/Select

Operational Scenario

Map Scenario

Players/Units to Test

Network Assets

Execute Test Event

Analyze Test Results

Compare Results to

Expectations Post Test

Analysis

Design Test/

Test

Preparation

Simulate/Preview

Test Event

67Reference: http://www.flickr.com/search/?w=all&q=castle+moat&m=text

68

Intrusion Detection

FirewallVPN Tunnel

• Compliance

• Standards / processes

• Service Level Agreements

• Defensive functions

• Offensive / exploitation functions

• Mission effectiveness

• Mapping cyber effects to mission effectiveness within cyberspace

• Mapping cyber effects to other warfighting domains

Categories of Cyber System Requirements

• International standards bodies:

• International Organization of Standardization (ISO)

• Payment Card Industry Security Standards Council (PCI)

• Information Security Forum

• The Open Group

• US-based standards bodies:

• National Institute of Standards and Technology (NIST)

• Federal Information Processing Standards (FIPS)

• DoD

• Department of Defense Information Assurance Certification and Accreditation Process (DIACAP) (and DITSCAP)

• Others (e.g., Joint Air Force Army Navy (JAFAN) , ICD 503)

Example Information Assurance Standards

• What is secure?

• Interpretation

• Death by standard

• Too many standards

• Conflicts

• Never underestimate the stupidity of people in large groups

• Typically seen as a low bar

Weaknesses

Polices, Standards, and Procedures

Governance Frameworks

• ISO

• SOx, PCI, HIPAA

• NIST, FIPS

• External

Policies

• High Level Guidance

Standards

• Implementation

Procedures

• Further Implementation

Implementation

• PCI-DSS 5: All systems storing credit card data must utilize anti-virus.Frameworks

• Policy-1: All systems will utilize with anti-virus,Policies

• Standard-1: All systems will utilize McAfee anti-virusStandards

• Procedure-1: All systems will utilize anti-virus with a given configuration.Procedures

Example

Risk Management Framework

(1) Categorize

(2) Select Security Controls

(3) Implement Security Controls

(4) Assess Security Controls

(5) Authorize System

(6) Monitor Security Controls

Data Owner

System Owner

System Security Office

•System Administrator

•Auditor, etc

Key People

System Security Plan

Security Control Matrix

System Security Office

Key Artifacts

Module 1.6 Information Assurance Standards 76

NIST 800-53 Control Families

ID Family Class

AC Access Control Technical

AT Awareness and Training Operational

AU Audit and Accountability Technical

CA Security Assessment and Authorization Management

CM Configuration Management Operational

CP Contingency Planning Operational

IA Identification and Authentication Technical

IR Incident Response Operational

MA Maintenance Operational

MP Media Protection Operational

PE Physical and Environmental Operational

PL Planning Management

PS Personnel Security Operational

RA Risk Assessment Management

SA System and Services Acquisition Management

SC System Communications Protection Technical

SI System and Information Integrity Operational

PM Program Management Management

Moderate-Low-Low Implementation Guidance

Identifier Family Class

Company Highly Confidential Baseline Controls

NIST 800-53 Full Control Family (455 Controls)

NIST 800-53 Control Tailoring

Not all baseline controls are appropriate for every system

Control may not be possible/feasible

Control may be overly burdensome

Control may not make sense

Tailoring process allows for system and risk specific implementation

Initial Security Control Baseline

Tailored Security Control Baseline

Data Owner Approved Set of Security Controls

Documented Agreed Upon Security Controls

(with rationale for any tailor in or tailor out)

Apply Tailoring Guidance

Data Owner

Acceptance

System Security Plan

Relationship of Risks to MissionT

hre

ats

/Vecto

rs

Information Assets

Thre

ats

/Vecto

rs

Countermeasures/Controls

Investment

Mission Threads/ Processes

Residual Risk

Assets

Summary of Network-Centric Measure Categories

• Interoperability

• Information Exchange Requirements (IERs)

• Message types/Formats

• Latencies

• C2/Planning timelines and effectiveness

• Information Security/Assurance

• Common Pictures (COP, CROP, CTP, SIAP…)

• Accuracy

• Number of tracks

• Timeliness

• …

• C&A - IA Compliance

• Controls/Countermeasure effectiveness

• Detection / Monitoring effectiveness

• Incident response effectiveness/timeliness and disaster recovery and business continuity

• Situation Awareness

• Ability to execute supported missions on networks/ information systems

Summary of Common Cyber Warfare Measure Categories

Test Results Mapped to Objectives

Obj # Summary of Objective

1.1Assess the ability of the SUT to detect attacks, probes and other CND events

1.2

Assess the SUT’s ability to manage, prioritize, filter and correlate CND related information and alerts from multiple disparate sources to distill voluminous detections into salient SA and detect threats otherwise undetectable

1.3Assess the ability of the SUT to store detailed data and allow for searches and queries to identify CND issues and past events

1.4Evaluate the SUT display to determine if necessary SA information is available to operators

1.5Assess the extensibility (ability to be modified and enhanced) of the SUT SA capability



2.1 Assess the ability of the SUT to detect emerging threats

2.2Assess the SUT’s ability to manage, prioritize, filter and correlate CND related information and alerts from multiple disparate sources to distill voluminous detections into SA on new and emerging threats

2.3Assess the ability of the SUT to store detailed information collected from multiple sources and allow for searches and queries to identify new and emerging threats

2.4Assess the ability of the SUT to display alerts and information so that operators and analysts can identify new and emerging CND threats

2.5Assess the extensibility of the SUT for collection, storage and analysis of CND information that can be used to identify and address new and emerging threats



3.1Assess the ability of the SUT to display information needed by the analysts and operators

3.2 Assess the usability of the SUT by the operators and analysts

3.3 Assess the sustainability and affordability of the SUT

3.4Assess the ability of the SUT to support the generation of administrative, summary and other reports

3.5Assess the interoperability of the SUT with other systems from the same vendor, other CND systems, and the enterprise network infrastructure as a whole

3.6Assess the ability of the SUT to facilitate responses to threats and attacks

Modification

Delivery

methods

Implementation

Reverse

Engineering

Discovery

Stealth &

Obfuscation

Approach /

Implementation

Shaping

Initial

Evaluation

Independent

Evaluation

Testing

As needed

Example Process:Vulnerability and Exploitation Process

Normal Dynamic Name Service (DNS) Operation

Web browser:

http://www.google.com

ISP’s DNS Server

Google’s Name Server

What is www.google.com?

What is www.google.com?

74.125.45.106

74.125.45.106

74.125.45.106

69.50.131.86

Poisoned Dynamic Name Service (DNS) Operation

Web browser:

http://www.google.com

ISP’s DNS Server

Google’s Name Server

What is www.google.com? 69.50.131.86

Attacker’s Name ServerAttacker

What is www.google.com?69.50.131.86

74.125.45.106

* On average, 2^16 attacks

Example Metric: (Beyond the basics)Duration of Attack Versus Conspicuousness

1

10

100

1000

0 100 200 300 400 500 600

Du

rati

on

(s)

[lo

gari

thm

ic s

cale

]

Packets Sent per Attempt

Duration vs. Packets Sent

The sweet spot-“Low and Slow”


• Threats

• Definitions



• T&E Challenges


• Planning



Outline

Basic Network Centric T&E Process

Define Objectives, Measures, and

Data Requirements

Define Test Network

Configuration

Generate/Select

Operational Scenario

Map Scenario

Players/Units to Test

Network Assets

Execute Test Event


Compare Results to

Expectations Post Test

Analysis

Design Test/

Test

Preparation

Simulate/Preview

Test Event

Define Objectives, Measures & Data Requirements

Plan and Design Test

Execute Test Events

Analyze and Report Data

Elements of a Net-CentricTesting Methodology

Network Centric Test System Infrastructure Drivers

• Decouple operational processes and scenarios from network specifics

• Facilitate development of new SUT interfaces

• keep pace with C4I and network systems

• Accommodate various intra-system communications modes

• support for tactical environments

Network Centric T&E Concepts

• Stimulation and Virtual Representations

• C4I System Interfaces

• Mapping the Virtual to the Real

• Intra-Test System Communications

• Test Execution


RISKACCEPT-

ABLE?

NO

DETERMINEOBJECTIVES

PRE-TESTANALYSIS

TEST EVALUATE

PRODUCT:KNOWN

RISKSOLUTION

IMPROVE

YES

Test and Evaluation

(T&E) Phases

Pre-testTest

Event

Execution

Post-Test

Objectives, Measures, and Data Requirements

Test objectives defined by the operational requirements

Measures and data requirements determined by user and tester

Generic Approach to Stimulation

InitiationMessage Type 32

ResponseMessage Type 35

Interactive Exchange

Type 33

Delay

between

messages

…

End Thread

Probabilities of branches

Completion of

Operational Task

• Background Traffic

• Interactions with Live

Players

Generate/Select Operational Scenario

Driven by operational objectives and measures

Operationally realistic

Live, virtual, or constructive

Simulation, scripted, or hybrid

Scenario generation tools

Simulation vs. Scripting Network/commtraffic (Sim vs. Stim.)

Simulation

Sophisticated battlefield simulations

Developed in training community

Large programs

Significant scenario development efforts

Often needed to assess or supplement “C2 effectiveness” measures with live forces

Scripting

Pre-built message database

Limited ability to adjust during an exercise

Simple and adaptable

More control of test events

Ideal for assessing IER measures

Ideal for generating “background” load

Host/Client-based

Virtual or bare metal

Generates end-to-end user traffic

Provides targets

Can represent threats

Packet/network-based

Packet streams sent through a network

Represents protocols

Does not typically represent full authentication, etc.

Does not typically represent a session properly

Can represent threat activity

Two Basic Types of Traffic Generation

Example Test Matrix

ID Description Threshold Data Requirements Form Event Sample Size

I-6 COI. Does the System allow the

embarked unit leader (Squad Leader

through Brigade Commander) to

command and control during

operations? [TEMP, para 2]

See App 6

to Annex

D

I-6 C-72 Critical Criterion (clarified).

PCLW. The vehicle's

communications system shall

provide for remote

transmission and monitoring

of any selected radio, and

for internal vehicle

communications (threshold).

App D-6,

paragrap

h 3

ALNO

Pending clarifications: 1) Must the SUT support an

"all-nets" broadcast over all radios in the

vehicle simultaneously from any one workstation?

2) Should each workstation operator be able to

monitor multiple radio nets simultaneously? If

so, what is the threshold number of nets? 3) Is

the second portion of the requirement, "and for

internal vehicle communications" redundant with

C-97 (ORD para 4.1.8.3.3.1)?

I-6 C-72 M-7 MOE. TD verification

that SUT can monitor

and transmit remotely

on any selected radio

from all

workstations.

Capability

Verified

TD Ver TD Ver AO/PT

AO/Comm

STE

NA

I-6 C-72 M-42 MOE. Percentage of

successful radio

access trials as

verified by VETT.

NFR Number of

successful radio

access trials

Form 27

VETT

AO/PT

AO/Comm

STE

36

Tri

als

I-6 C-72 M-43 MOE. Battalion Staff

ratings of the SUT's

capability for remote

transmission and

monitoring of radios.

> 50%

question

s with

>= 80%

favorabl

e

response

s

PTS Q#: 176, 177,

619

PTS AO/Comm

STE

AO/DegL

STE

AO/OMP 1

AO/OMP 2

AO/OMP 3

9

sol

ide

rs

Example Information Operations Attack System

System Under

Test Client

Communications

Node (CN)

PTT

PTT

VMF, C2PC, HTTP,FTP,

OTH Gold, VoIP

Communications

Node (CN)

Node Control Console

(Test Operator)

Radio Nets

IP Networks

Test and Engineering Network

System Under

Test Client

Systems Under Test

STARSHIP

Basic Control

Video Conferencing

Direct Injection

Client/Server

PTT

PTT

PTT

PTT

VMF, C2PC, HTTP,FTP,

OTH Gold, VoIP

IP or PSTN

Networks

PTTPTT

Voice/Video Emulation Test Tool (VETT)

Typical System Configuration

Two Nodes Plus Controller

Attacks

Denial of Service Attack

IRC Server (NI)

Wircd.exe

Bot Mater (I)

Subprogram Agent

launch_attack.exe

Image Get (I)

HTTP Agent

(Emulates Web

User)

Bots (I)

Subprogram Agent

gspot.exe

Web Server (I)

Subprogram Agent

LightTPD.exe

1.5 Mbps Link

LAN

Legend:

( I ) – Instrumented

( NI ) – Not Instrumented

LAN

IRC Channel(Botnet Control)

Results: Web Service Delay Cause by Attack

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

0.0

0

0.3

5

0.7

0

1.0

5

1.4

0

1.7

5

2.1

0

2.4

5

2.8

0

3.1

5

3.5

0

3.8

5

4.2

0

4.5

5

4.9

0

5.2

5

5.6

0

5.9

5

6.3

0

6.6

5

7.0

0

7.3

5

7.7

0

8.0

5

8.4

0

8.7

5

9.1

0

9.4

5

9.8

0

10

.15

10

.50

10

.85

11

.20

11

.55

11

.90

12

.25

12

.60

12

.95

13

.30

13

.65

14

.00

14

.35

14

.70

Min

seco

nd

s

Retrival Time

Processing Delay

Attack Bounds

• Joint Information Operations Range (JIOR or IO Range)

• Joint Mission Environment Test Capability 2.0

• National Cyber Range

• Lab ranges

• Some connected to JIOR

• Built from virtualized systems and networks

• Provide various traffic generation capabilities

• Threat representation / Red Teams

• Provide teams to attack systems

• Generally connected to the JIOR

• Examples:

• AF 346TS, Lackland AFB

• USN SSC PAC – Pearl City, HI

• USA Threat Systems Management Office (TSMO)

Cyber / IO Ranges

Typical Range Set Up: Attacks Through Gray Space

Servers

Endpoint Devices

Malware

Tools

Attacker

Endpoint

Servers

Defenses

Gray Space

Target Space orBlue Space

Malicious Actor Space orRed Space

User

Pivot in GraySpace

• Gray space is neutral networks/machines• Used to hide tracks• Often in countries other than target or attacker


• Threats

• Definitions



• T&E Challenges


• Planning



Outline

Plan and Design Test

Define Test System Configuration


Map Virtual Players/Units to Test Network Assets


Based on measures and data requirements

Stimulation and data capture

Distributed, Undistributed, or Hybrid Approach

Data collection management

Time synchronization

Generic Approach to Stimulation

InitiationMessage Type 32

ResponseMessage Type 35

Interactive Exchange

Type 33

Delay

between

messages

…

End Thread

Probabilities of branches

Completion of

Operational Task

• Background Traffic

• Interactions with Live

Players

Email(Thread Initiation)

…


Text Message(End of Thread)

VoIP Phone Call

Chat Session

…

Image Reference: http://www.grime.net/facets/air.htm

VMF via SADL(End of Thread)

C2 Message

Live Player(End of Thread)

…

Mapping the Virtual to the Real(Decoupling operations for IT specifics)

• Assign “operational” roles to real-world network assets

• Rapidly reconfigurable-Adapt to changes in networks

• Accommodate mixtures of Live/Virtual/Constructive

• Reuse of validated “scenarios”

G2 Free form

Intel messages

S3 Situation Reports

G3 Voice

Email Client

Email@address

GCCS COP Server

IP address

JVMF Message Process

Universal Reference Number

VoIP Call Manager

Phone number

…

Execute Test Events

Control scenario events and threads

Data aggregation

Time synchronization controls


Two methods: Direct injection and C2 application/system stimulation

C4I and Cyber System Interfaces

Direct Injection on Network

Must validate messages/formats

Protocol/login/encryption issues

Streamlined approach –typically yields more volume

C2 Application/system stimulation

End-to-end system test!

Validation “extends” from C2 app/system

Protocol/login/ encryption handled naturally

Volume can be limited by C2 app/system

Verification & Validation Issues

• Correct network load/data

• Broadcasts, unintended services, etc.

• Message format verification

• Protocol verification

• Security certification

• Scenario Validation

Intra-T&E System Communications

• Dedicated T&E “out-of-band” network

• T&E monitoring/control transparent to SUT

• Connectivity using SUT networks

• Limited use of bandwidth

• Use often limited to pre and post-test event

• Disconnected mode (“Sneaker” net)

• Hybrids - Use of all three modes simultaneously

Practical Issue: Time Synchronization

• Synchronizing test system

• Synchronizing SUT

• GMT or local

• Use test network or SUT network

• Time sync section in test plan/procedures

• Synchronization checks on daily check list


• Threats

• Definitions



• T&E Challenges


• Planning



Outline



Compare Results to Expectations

0

20

40

60

80

100

120

140

160

180

200

Task

Ex

ec

uti

on

Tim

e (

se

c)

Multi-ship

Geolocation

CAS Mission Time Sensitive

Targeting

End-to-end

track update

Planning

Pre-contact

Contact

Post-contact


• System Logs

• Network Devices

• Defensive Devices

• Sniffers/Protocol analyzers

• Management system reports

• Voice / Video

• Modeling and simulation/stimulation

• Command and Control Systems

• Cyber / IO ranges control

• Red team logs / penetration test reports

• Electro-magnetic environment

Types of Data Sources

• Leverage SUT for instrumentation

• Leverage existing instrumentation within environment

• Add instrumentation to validate data

Instrumentation Tricks

Data Reduction/Analysis Concepts

• Systematic tie-in with Simulation/Stimulation and Instrumentation

• Identifiers (or serial numbers) to associate inputs (sent messages) with outputs (received)

• Database driven approach allows for recalculation of measures for different requirements

Data Reduction/Analysis Concept (Continued)

• Automated and rapid data reduction

• Need for speed!

• Quick look analysis during test execution

• Allows tester to adapt between test events

Cyber T&E AnalysisProblem Definition:

Finding the “needles” in the

digital haystack

• Agile C2 concepts, particularly at the Joint Operational Level of warfare, loosely define operational task threads

• TTPs can change as an operation is conducted

• Numerous applications and protocols (VoIP, chat, http, email, etc.) can be used to accomplish tasks

• Threat representations can be difficult to instrument and process

• Heterogeneous network traffic is voluminous

• Operationally significant transactions are “needles” in the digital hay stack

• Tester must track network operations in near-real-time

• Ensure the right data/events are being collected

• Adjust test plans as operation dictates

• Quick look analysis

Cyber T&E AnalysisProblem Definition:

• Distinguishing threat activity from benign activity

• Tracking and “profiling” threat sequences

• Correlating network data with red team logs

• Reconciling the “good guys” view from the red team view

• What really happened


• Threats

• Definitions



• T&E Challenges


• Planning




Outline

Technologies for

Cyber Testing

Technologies for Testing in Cyberspace

• Building the environment

• Threat Representations

• Instrumentation and

situation awareness (SA)

• Data reduction and analysis

Parallels between operational requirements and testing requirements!

Building Representative Environments: Key Technologies

• Virtual machines and networks (VMWare®, Citrix XenServer™)

• Replicate user machines and LANs on server farms

• Lower cost/ footprint

• Numerous open source tools for threat and defense representation (nmap, metasploit, snort…)

• Nessus and other commercial vulnerability scanners

• Ranges (IO Range, National Cyber Range)

• Varying levels of classification

• Numerous facilities and capabilities for testing on the range

• Connectivity to “open air” (non-virtual)

• Traffic generation (benign and threat)

• Generation server to generation server

• Represent traffic on the wire

• High volumes

• End-user machines

• Represent attacks on targets (bare metal or VMs)

• Lower volumes

• Threat “teams” and Penetration testers with various specialties

• Each service has team(s) and some Joint teams

• Specialties may include IP-based, SCADA, C2 systems, social engineering, etc.

Gap: Up-to-date, faithful threat representations

Traffic generation and threat representation

Ingest Processing Threat Reps/Automated Intelligence

Malware

Web

Presence

Social

Engineering

Open

Source

Network

monitor

Closed

Sources

Plug-able

Architecture

Traffic

Generators

Threat Actor Characteristics

Wide Array of

Open and Closed

Intelligence Inputs

Correlation &

Profiling

Threat

Representations

Automated Cyber Threat Representations (ACToR): Project Overview

135

Capability Summaries

Engagement Guidance

Cyberspace Data Reduction and Analysis Requirements

Measures for “Agile Cyber Warfare”: Timely and effective

decisions

Methods to nail down effectiveness/operational impacts

Technology needed to “mine” measures from

heterogeneous and voluminous network traffic

Correlation of events on disparate and distributed media

Tracking and making sense of cyber warriors’ agility

Timeline (SV-10c) Visualization

Log File Event

Node-node transactions

Color-coded type(e.g., https, http, ssh)

Mouse hover over transaction or log pops

up more detailNode (e.g., server, client,

application, service)

High light threat activity

Timeline Visualization (Continued)

Select TCP Payload to get transaction

message

Transaction message Popup

Copy Payload and paste into file

StatusMap

Mouse over headingto get full description

Mouse over demo caseto get associated task

Mouse over heading, then click on + to expand view into heading specific items

Inspect data for selected run time

Select run-time of interest

Documents

Test and Evaluation of Cyber Systemsitea.org/images/pdf/conferences/2015_Symposium... · networks • Hackers may be multiple motivations, including profit, protest, or because of