Applying Big Data Analytics in T&E - Home - ITEAitea.org/images/pdf/conferences/2015_Symposium... · 2015. 8. 24. · DRAFT UNCLASSIFIED –DISTRIBUTION STATEMENT A Reference Number

DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A

Reference Number 15-S-2335; 08-14-2015

Applying Big Data Analytics in T&E

August 2015

Ryan Norman

TRMC Initiative Lead for Knowledge Management

[email protected]

mailto:[email protected]


Reference Number 15-S-2335; 08-14-2015 2

Source: IDC’s Digital Universe Study, sponsored by EMC, Dec 2012

More information over the next two years than in the entire history of mankind!

Name (Symbol) Value

2020 40,000 EB

2005 130 EB

2010 1,250 EB

2015 8,000 EB EX

AB

YT

ES

Worldwide Exponential Growth of Data



Test & Evaluation Growth of Data

• Larger Test Footprints–4-on-4 test flights (more systems per test)

–Much faster weapons systems

–Geographic separation not as effective as it

used to be

• Demand for Shorter Acquisition

Cycles–More concurrent testing

–More real-time analysis

• Increased System-of-Systems

Test Complexity–“Five Futures” (EW, UAV, NCO/W, DE,

Hypersonics)

– Integrated fleet (F-18E/F, E-18G, F-35, SM

VI, UAV)

–“Swarming” UAVs

Increased System Complexity

Integratedweapon system(1-20 Mbps ea.)

Integratedavionics(2-10 Mbps)

Separationvideo(1-8 Mbps)

Cockpitvideo(1-5 Mbps)

Integratedcommunications(0.5-2 Mbps)

Flight test transducers(0.5-10 Mbps)

Multi-spectralsensors(1-12 Mbps)

Integrated self-defense system(0.5-2 Mbps)

Total Throughput: 7.5Mbps – 70Mbps+

3

T&E Mission: Acquire data and discern into knowledge

44

Big Data / Knowledge Management (KM)Challenges & Needs

T&E Infrastructure Challenges:

– How do we conduct T&E of increasingly complex, data-driven systems?

– How do we enable more efficient & continuous system evaluation?

Need: A DoD-wide KM capability for T&E to help

achieve better acquisition outcomes and reduce costs

– Trusted processes across government and

industry that identify problems sooner rather than later

– Accessibility of knowledge & data to legitimate users

– Discoverability of knowledge & data obtained

over time

– Availability of knowledge through common tools &

technologies – including DoD T&E cloud solutions

– Leverages proven Industry techniques / practices

Big Data Analytics depends on effective Knowledge Management



The TRMC “Blueprint”:Putting Test Capabilities on the DoD Map

Risk mitigation needsTechnology shortfalls

Risk mitigation solutions Advanced development

Capabilities

Service Modernization and

Improvement Programs

Acquisition Programs and Advanced

Concept Technology Demonstrations

T&E Multi-

Service/Agency

Capabilities

DoD Corporate

Distributed Test

Capability

TRMC Joint Investment

Programs

Transition

Requirements

Strategic Plan for

DoD T&E Resources

Annual T&E

Budget

Certification

(6.3 Funding) (6.6 Funding)(6.4 Activity)

DT&E / TRMC

Annual Report

Defense Strategic Guidance

Service T&E Needs and Solutions Process

Acquisition Process

5

6UNCLASSIFIED – DISTRIBUTION STATEMENT A

Realizing Improved DoD T&E Knowledge Management

1. Understand and Document T&E challenges & needs

– (FY12) Completed Data Management for Distributed Testing (DM-DT) Study− Result: Developed functional requirements for T&E enterprise distributed Data Management

– (FY13) Comprehensive Review of T&E Infrastructure report published− Key Recommendation: Use DoD cloud solution for T&E data

− Key Recommendation: USD(AT&L) establish a DoD-wide KM capability for T&E to help achieve

better acquisition outcomes and reduce costs

2. Execute proofs of concept that inform an enterprise approach to T&E

Knowledge Management

– (FY14-15) Joint Strike Fighter Knowledge Management (JSF-KM) project− Goal: Assess KM technologies and methodologies in support of an existing acquisition program

– (FY15-16) Collected Operational Data Analytics for Continuous Test & Evaluation

(CODAC-TE) project− Goal: Apply KM technologies and methodologies across the lifecycle

3. Develop investment plan that achieves strategic objectives:

– Integrate T&E infrastructure into cohesive Knowledge Management enterprise

– Modernize T&E practices & processes to leverage Big Data analytics techniques

– Apply Big Data analytics tools & techniques to the T&E mission space



JSF Big Data Analysis Challenges

• More data is being collected for JSF than can be quickly

analyzed

– JSF-KM Deliverable: Big Data analysis tools applied to JSF test data

• Data sets are too large to download, requiring engineers and

analysts to travel

– JSF-KM Deliverable: Remotely accessible Knowledge Management

system installed at Edwards and Nellis

• Many data sets are not well organized

– JSF-KM Deliverable: Knowledge Management portal that catalogs,

organizes & retrieves JSF data across selected test locations

• Utility of data limited by our ability to interpret & use it

– JSF-KM Deliverable: Big Data visualization tools applied to JSF test data



User

Eglin

Pax

River

Joint Strike Fighter Knowledge Management (JSF-KM) Test Concept

• DT & OT data storage in government facilities

• Collect / Store more precise data during OT

• Search / Analyze Edwards & Nellis

data from any secure location

• Bring enhanced JMETC infrastructure to JSF T&E

• Apply commercial Big Data and Knowledge Management tools to DoD requirements

• Knowledge shared across

JSF DT / OT T&E locations

• Scalable to other JSF T&E locations

8

JPO

Pote

ntia

l Expansio

n

UNCLASSIFIED - Distribution A

JSF-KM Improvements to Existing T&E Capabilities

9

DT Today

OT Today

With JSF-KM

Parallel

Data

Ingest

30 minutes

(multiple aircraft)

Raw Data

Available

Video/Data at Post-

Mission Debrief Big Data

Analytics

Govt. Analyst

Data Request

Analysis

Note: Numbers reflect single 2 hour flight mission

Data

Ingest Raw Data

AvailableGovt. Analyst

Data RequestAnalysis

2 hours

(per aircraft)1 day 1 week

30 seconds

Data

Ingest Raw Data

AvailableGovt. Analyst

Data RequestAnalysis

1-2 hours

(per aircraft) 10 minutes 4-5 hours

Data Ready for

Use @ (Govt)

30 seconds

30 seconds

Data Ready

for Use @ LM

>20 weeks of data

available online

Data Ready for

Use @ (Govt)3 weeks of data

available online



JSF-KM Success Stories October 2014 – May 2015

• Resolved test data time correlation issues

– Time stamps in data files found to be corrupted post-mission

– JSF-KM analysis tools were able to correct the time correlation issue

– Without JSF-KM, at least five missions would have been re-flown

• Video available during post-mission debrief due to JSF-KM data

ingest improvements from DART Pod

– Existing tools could not process video in time to support post-mission de-brief

– Without JSF-KM, there would be no flight video during post-mission debrief

• Discovery of avionics box issue

– Pilot and Analyst discovered problem from video data available 30 minutes after landing

– Avionics Box was replaced before another mission was flown

– Without JSF-KM, problem would not have been discovered for several days

10

Return on Investment has been realized before

deploying any Big Data analytics capabilities



Collected Operational Data Analytics for Continuous Test & Evaluation (CODAC-TE) Proof of Concept

• Background: US Army Aberdeen

Test Center (ATC) has ~30TB of

underutilized data – including 20TB

of in-theater operational data

• Goal: Utilize Big Data analytics

across multi-commodity DT, OT,

and in-theater system data to

discover “unknown unknowns” for

current and future Army systems

• Use Cases: Mine-Resistant Ambush

Protected (MRAP) Theater Data;

Camouflage Effectiveness

• Leverages High-Performance

Computing Major Shared Resource

Center and ATC expertise

Challenges being addressed:

• Insufficient data science expertise

• Current analytical systems inadequate for today’s data volume and velocity

• Lacking tools and techniques for discovering unknown unknowns and conducting

complex trends analyses



Knowledge Management (KM) Initiative Summary

• TRMC is acting upon the KM recommendations from the

Comprehensive Review of T&E Infrastructure. Strategic Goals:

– Integrate T&E infrastructure into cohesive Knowledge Management enterprise

– Modernize T&E practices & processes to leverage Big Data analytics techniques

– Apply Big Data analytics tools & techniques to the T&E mission space

• TRMC-funded proofs of concept will deliver proven capabilities

– Enable Big Data analytics during JSF T&E

– Improve transfer of knowledge between fielded and next-gen systems

– Inform T&E investment plan that advises future infrastructure, process, and

workforce decision-making

• Improved T&E KM will help achieve better acquisition outcomes and

reduce costs

– Identify & Diagnose problems sooner and continuously

– Inform acquisition decisions through larger knowledge base

– Achieve T&E infrastructure efficiencies

12



Questions?

Ryan Norman

TRMC Initiative Lead for Knowledge Management

[email protected]

13

mailto:[email protected]


What is Knowledge Management (KM)?

• Data represents a fact or statement without relation to other

things

– Example: It is raining

• Information is data that has been placed in a meaningful

context

– Example: The temperature dropped 15 degrees & then it started raining

• Knowledge is a collection of information with some extracted

value

– Example: If the humidity is very high & the temperature drops substantially, the

atmosphere is often unlikely to hold the moisture – so it rains

• Knowledge Management (KM) is the process of efficiently

capturing, handling, distributing, and using knowledge

• KM is based on two critical activities:

– The capture & documentation of explicit & implied knowledge

– The dissemination of this knowledge within an organization(s)



Long-term DoD T&E Enterprise Knowledge Management Vision

15

Result: T&E data used more effectively & efficiently during acquisition

User• The primary product of T&E is data & knowledge

• Embrace KM & Big Data Analytics to efficiently handle & securely share T&E data

• Organize T&E data to build knowledge across all DoD acquisitions

• Federate distributed data repositories to enable execution & automated search scenarios that cannot occur today

• Utilize modern mechanisms to enable collaboration between SMEs in government and industry


A Selection of Items Under Assessment

• Process:

– What business process changes to T&E are needed to enable an enterprise

approach?

– What best practices and capabilities from industry are applicable to T&E?

– What best practices and capabilities from the Intelligence Community are

applicable to T&E?

• Technology:

– What costs more overall: processing power or network bandwidth?

– More efficient / effective for T&E: large-scale crawl and index (eg. hadoop) or

targeted domain-specific intelligence?

– How can we leverage existing infrastructure investments in supercomputing

and knowledge management?

• Relationships:

– What are the “touch points” for T&E knowledge across acquisition?

– How to we promote trust and knowledge sharing with and among Industry?



Joint Strike Fighter Knowledge Management (JSF-KM) Effort Overview

• TRMC is investigating tools, techniques, resources, policies &

procedures needed to more efficiently & effectively use T&E data

• TRMC is partnering with Joint Strike Fighter (JSF) to ascertain

how Big Data tools & Cloud Computing technologies could assist

acquisition programs during T&E

• TRMC will establish a next-generation Knowledge Management

(KM) capability that utilizes the latest in virtualization

technologies, methodologies, & best practices for JSF OT data

• Capability will enable remote search & retrieval of JSF OT&E data

17

JSF JPO is assisting TRMC in prototyping

a Big Data / Enterprise KM system

Documents

Applying Big Data Analytics in T&E - Home - ITEAitea.org/images/pdf/conferences/2015_Symposium... · 2015. 8. 24. · DRAFT UNCLASSIFIED –DISTRIBUTION STATEMENT A Reference Number