Mr. David Jimenez, ETD

US Army Test & Evaluation Center

For All Audiences

April 2015

U.S. Army Test and Evaluation Command

US ARMY TEST AND

EVALUATION COMMAND

Supporting a Leaner, More Capable

Expeditionary 2025 Army

Mission, Vision, End State

MissionATEC plans, integrates and conducts experiments, developmental

testing, independent operational testing and independent evaluations and assessments to provide essential information to acquisition decision makers and commanders.

VisionTo be a team of highly skilled test and evaluation professionals

focused on informing equipping decisions for today’s and tomorrow’s warfighter.

End StateThe Army possesses a versatile mix of adaptable Soldiers and units

equipped with proven materiel based on decisions informed by independent evaluations of technically proficient and versatile experimental, developmental, and operational tests.

2

Where We Are

Ft GreelyCold Regions Test Center

Ft Huachuca

Ft Hood

Ft Sill

YumaProvingGround

Aberdeen ProvingGround

HQ ATEC

White SandsMissile Range Legend

Headquarters

Test Center

Operational Test Dir

PanamaTropic Test Site

AEC

Ft Bliss

DugwayProving Ground

Electronic Proving Ground

RedstoneArsenal

Ft Bragg

OTC

3

FY15 Assigned Personnel (8,279)*

504 Military

3664 Civilians

4111 Contractors

* as of 28Feb15 AEC – Army Evaluation Center OTC – Operational Test Command

4

= ATEC

Nevada Test and Training Range

Utah Test and Training Range

West Desert Test Center

Cold Regions

Test Center

NAWC-WD China Lake

Electronic Proving Ground

30th Space Wing

Keyport

NAWC-WD

Point Mugu

412th Test Wing

Yuma Test Center

Tropic Regions

Test Centervarious locations

US Army Kwajalein Atoll

Reagan Test Site

PMRF DISA, JITC

White Sands Test Center

96th Test Wing (includes 96th Test Group)

Atlantic Undersea Test andEvaluation Center

NAWC-ADPatuxent River

Arnold EngineeringDevelopment Complex

45th Space Wing

AberdeenTest Center

DISA

Legend:

Army Navy Air Force Defense Agency

Major Range and Test Facility Base(MRTFB*)

23 Sites – Army 8, Navy 6, Air Force 7, Defense Agency 2

*DoDD 3200.11 and DoDI 3200.18

ATEC Organizational Chart

ATEC: Army Test Evaluation Command

OTC: Operational Test Command

AEC: Army Evaluation Center

OTA: Operational Test Agency

WDTC: West Desert Test Center

WSTC: White Sands Test Center

ATC: Aberdeen Test Center

DPG: Dugway Proving Ground

EPG: Electronic Proving Ground

RTC: Redstone Test Center

WSMR: White Sands Missile Range

YPG: Yuma Proving Ground

BMD OTA: Ballistic Missile Defense Operational Test Agency

CRTC: Cold Regions Test Centers

TRTC: Tropical Regions Test Center

YTC: Yuma Test Center

IEWTD: Intelligence Electronic Warfare

Test Directorate

CG/Staff

/

BMD OTA

Redstone

AEC

/ ATC RTC YPG

• CRTC

• TRTC

• YTC

DPG

• WDTC

WSMR

/ OTC

/

*O/O IEWTD

(OPCON)

• WSTC

General Officer

Senior Executive Service

5

EPG

• IEWTD

*On Order IEWTD OPCON to OTC for Operational Test missions

Changes to Test and Evaluation

Policies

Stemming from DODI 5000.02 affecting

Army future programs and 2025

execution

6

ATECLEAD OTALEAD DT&E

ORGANIZATION

DODD 5000.01

“Each Military Department shall

establish an independent OTA

reporting directly to the Service

Chief to plan and conduct

operational tests, report results,

and provide evaluations of

effectiveness and suitability.”

DODI 5000.02 and Title 10

(A) providing technical expertise on testing and evaluation issues to the chief developmental tester for the program;

(B) conducting developmental testing and evaluation activities for the program, as directed by the chief developmental tester; and

(C) assisting the chief developmental tester in providing oversight of contractors under the program and in reaching technically informed, objective judgments about contractor developmental test and evaluation results under the program.

INDEPENDENT

FROM PM &

SUPPORTS

DECISION MAKERS

SUPPORTS

MATERIAL

DEVELOPMENT

AND

MATURATION

ATEC Has Two Roles

8

DODI 5000.02 Changes Impacting ATEC

Change in DODI 5000.02 Change to ATEC Process/Product

Shift Left - Emphasis on Developmental Test

and Evaluation (DT&E)

System Evaluation Plans and Test & Evaluation Master

Plans (TEMP) will include both:

• Developmental Evaluation Framework (DEF)

• Operational Evaluation Framework (OEF)

TEMP required at Milestone A

• Update at each Acquisition Milestone

• Early Strategy Review / Concept In-Process Review

replaced with Test & Evaluation Strategy Review (TESR)

• Approve TEMP inputs at each milestone

(TESR A, TESR B, etc)

Lead DT&E Organization reports to Chief

Developmental Tester at the PM

Continuous Evaluation publication will be used as

reporting mechanism for DT&E

Operational Test Agency Assessment of

Concept of Operations T&E Implications

New ATEC input into the TEMP at Milestone A

T&E Reports and Supporting Data and Meta-

data to DTIC

New requirement to provide supporting data and meta-

data in addition to the T&E Reports

Addition of a Developmental Request for

Proposal release milestone decision for

Engineering and Manufacturing Design (EMD)

and often for Low-Rate Initial Production (LRIP)

Will drive additional and more frequent input to PM for

acquisition strategy, test schedule, cost estimates, etc.

(TESR, SEP-U, CE PUB will help)

9

Change in DODI 5000.02 Change to ATEC Process/Product

Shift Left - Emphasis on Developmental Test

and Evaluation (DT&E)

System Evaluation Plans and Test & Evaluation Master

Plans (TEMP) will include both:

• Developmental Evaluation Framework (DEF)

• Operational Evaluation Framework (OEF)

TEMP required at Milestone A

• Update at each Acquisition Milestone

• Early Strategy Review / Concept In-Process Review

replaced with Test & Evaluation Strategy Review (TESR)

• Approve TEMP inputs at each milestone

(TESR A, TESR B, etc)

Lead DT&E Organization reports to Chief

Developmental Tester at the PM

Continuous Evaluation publication will be used as

reporting mechanism for DT&E

Operational Test Agency Assessment of

Concept of Operations T&E Implications

New ATEC input into the TEMP at Milestone A

T&E Reports and Supporting Data and Meta-

data to DTIC

New requirement to provide supporting data and meta-

data in addition to the T&E Reports

Addition of a Developmental Request for

Proposal release milestone decision for

Engineering and Manufacturing Design (EMD)

and often for Low-Rate Initial Production (LRIP)

Will drive additional and more frequent input to PM for

acquisition strategy, test schedule, cost estimates, etc.

(TESR, SEP-U, CE PUB will help)

Impacts to ATEC

• DEF required for ACAT-I and

recommended for all others.

• AEC provides major input to DEF.

• Separate table/section in TEMP

• SEP format change required to maximize

use of DEF tool.

• Advantage of providing the TCs clearer DT

guidance.

• DEF working efforts included for WIN-T

Inc3, AIAMD

10

DODI 5000.02 Changes Impacting ATEC

Developmental Evaluation Framework

Test and Evaluation Master Plan (TEMP) includes a Developmental Evaluation

Framework (“T&E Roadmap”)

―Knowledge gained from testing provides information for

technical, programmatic, and acquisition decisions. DoDI 5000.02 (Interim)

Developmental Evaluation Framework:– Identifies key data that contributes to assessing progress on:

− Key Performance Parameters

− Critical Technical Parameters

− Key System Attributes

− Interoperability requirements

− Cybersecurity requirements

− Reliability growth

− Maintainability attributes

− Developmental test objectives

− Others as needed

– Show the correlation/mapping between:

− Test events

− Key resources

− Decision supported

11

Increased T&E Activity at Early Stages in the

Life Cycle Management System

BA C

Materiel Development

Decision

CDD Validation

Dev RFP Release

Low-Rate Initial

Production (LRIP)

Initial Operational

Capability (IOC)

Full-Rate Production (FRP)

Decision

Full Operational Capability

(FOC)

Materiel Solution Analysis

Technology Maturation &

RiskReduction

OT&E, Production

and Deployment

Engineering & Manufacturing Development Sustainment Disposal

DoDI 5000.02

Initial

Capability

Document

Capability Based

Assessment(CBA)

AEC Req’tOps Concept

Analysis

Developmental Test Planning/Execution

& EvaluationOperational Test

Planning & Evaluation

TEMP

T&E WIPTCoE Experiment

Support

TEMP-U TEMP-U TEMP-U

EOA OA OA

EOA = Early Operational Assessment

OA = Operational Assessment

T&E WIPT = Test & Evaluation Working

Integrated Product Team

CoE = Center of Excellence

TEMP/U = Test & Evaluation Master

Plan/Update

Continuous

Data Collection

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

Pre Nov 13 Current

Manye

ars

Delta

+370% +20% +11% +6% +1%

DEF

OEF

TESR A TESR B TESR C TESR FRP

SEP SEP-U SEP-U

12

ATEC: Reducing Risk in 2025

through Test & Evaluation

Optimizing T&E Data

• Obtaining and Retaining Information

• Rapid Analysis

• Greater Breath of Analysis

• Advanced Cybersecurity T&E

iknnn

i

ttinnnttn

ikn

i

ttintn

ikn

i

ttin

ikn

i

ttin

ikn

i

ttin

eeinnni

nnneen

eeini

ne

eeini

n

eeini

n

eeini

ntR

1544

22544335

44

22435

33

32323

22

32322

11

111

0

1)(

544

544)1(

5

0

1)(

4

4

0

1)(

3

3

0

1)(

2

2

0

1)(

1

1

!!

!1

!!

!

!!

!

!!

!

!!

!)(

Gartner’s 2015 Trends:

#4 – Advanced,

Pervasive, and Invisible

Analytics

16

Effectiveness, Suitability & Survivability

(ESS) Assessment and Reporting Policy

Overall two-tier effectiveness, suitability and survivability

Key Parameter Summary for XX System

KPP 1: Range (m)

KSA 2: Reliability

(MRBBEFF – Hrs

80% LCB)

0 150

KSA 1: Lethality (See

Classified Report)

KPP 2: Accuracy (m)20 0

KPP 3: Operational

Availability (%)0 100

90

KSA 3: Fire Mission

Time (sec) 180 0

150Eff

ecti

ven

ess

Su

itab

ilit

y

Legacy XX System Requirement+

5

0 8500

7390

Greater than YY system

7500

15 4

60

32

92 97

Requirement

Less Capability More Capability

8465

120

86 143

Builds upon AoA analysis conducted

by AMSAA & TRAC.

Intent is to show decision makers how

key system parameters compare to

the requirement and legacy systems.

Continuous scales convey greater

information than binary met/not met.

Scales need to be adjusted so less

capability is always on the left and

greater capability on the right. 0

should almost always be included on

each scale to limit the likelihood of

“lying with statistics”.

Depending on the program, assessed

numbers should be stated with

confidence (e.g. 80% LCB) as much as

possible for both legacy and new

systems.MRBEFF (80% LCB)

13

Operational Impact Rating Provides

Further Analysis of Test Result Impact

This is a “reasonableness” evaluation. For example• Minimum speed: test result = 53 mph. Requirement = 55 mph

• Requirement rating: could be yellow or red based on the full context of

the evaluation.

• Operational Impact: could be green based on the system CONOPs,

current system performance, etc.

Rating Symbol Operational Impact Rating Definition

Similar or

Enhanced

Capability

The system evaluation finding indicates that the operational capability is

similar to current capabilities, provides an improved capability, or provides

a new capability, relative to the operational task.

Reduced

Capability

The system evaluation finding indicates that the system may result in

decreased mission capability, relative to the operational task.

Significantly

Degraded

Capability

The system evaluation finding indicates that the system may have

significant, negative impact on mission capability, relative to the

operational task.

Unknown The system impact on mission operations is not known and cannot be

determined from the information and data available.

14

15

Summary of Cyber Security Risk Metrics and

Mapping to AEC DocumentsLR DHS Entities to examine Condition exploited by

attacker

Input to operational metric

1 HWAM Subnets and

removable storage

Unauthorized Devices Authorized device on subnet and authorized removable storage on devices; list of first and last

seen times of unauthorized devices, attachment logs for removable storage

2 SWAM Software on devices Unauthorized Software Un/Authorized software on each device. List of first and last seen times for each unauthorized

software package on each device, anti-malware findings

3 VULN Vulnerabilities on

devices

Known vulnerabilities Vulnerabilities for which we need to scan. List of first and last seen times and score for each

vulnerability on each device

4 CSM Configurations on

devices

Mis-configurations Tests (including expected results) on each device. List of first and last seen times and score for

each failed test on each device

5 TRUST Role assignments to

users

Roles assigned to

untrustworthy user

Role assignment to users providing access to assets, asset theft model, vetting history of users

6 CRED Role assignments to

accounts

Roles assigned to

compromised accounts

Role Assignments to Accounts providing access to assets, asset theft model, authentication

policy for accounts

7 PRIV Permissions and role

assignments to

accounts

Permissions assigned to

accounts and risk

incurring roles

Permissions that should be assigned to an account for each role assigned to the account.

Permissions assigned to each account, risky roles for each account

8 BEHAVE Users education Users lacking training or

education

Required education for each role a user hold

.Education history for each user relative to assigned roles

9 BOUND-N Network topology and

filtering

Overly permissive

reachability

Network topology and filtering and a exploitation model for protocols

10 BOUND-P Access to physical

spaces by user and

mechanism

Physical access to cyber

assets

Mapping of the resources of devices from LR-5, LR6, LR-7, mapping of devices to physical

areas, account usage pre devices, access control policies and assignments to physical areas

11 BOUND-V Encryption of

resources on devices

Lack of encryption

enabling resource theft

Inputs of LR-10 together with encryption status of resources and devices

SUT Environment

2025 T&E and Big Data

16

Goals:• Utilize knowledge, information, and data to

achieve core mission and business objectives.

• Faster, more Accurate Decision Making

• Cost Optimization

• Quicker Responses to Requests for Information

• More Holistic Test and Evaluation

• Automated tracking items or status

• Make useful big data capabilities available to everyone, but tailored to specific needs.

Sustainment of data for long term use

(Archival)

Discoverability and Access to data

Analytics of historical and current information

Derive context to inform decision

making

Common Core Requirements:

2025 T&E

Leveraging Historical Data

Faster, More Sophisticated

Analytical Tools

Modeling & Simulation

Design of Experiments

Cloud Computing

Conclusion

• Emerging DoD policies requires

in test and evaluation planning.

major risks earlier / increase likelihood

of successful program execution.

(two-tier) evaluation criteria provide

better insight to decision makers.

leveraged to the greatest extent possible

to provide content based analytic and design

insights.

18