Continuous Transformation Environment

Phase III Implementation, Prototypes, and T&E-based

Use Cases

Overview

• Phase III Technical Goals• Initial CTE Technology Focus Areas• Overview of the CTE Technology ingest

process• Use-Case Approach to T&E• GPU Investigations• Current CTE Use Cases• Next Steps

2

Phase III Technical Goals• Establish a large-scale testbed capabilities that enable emulation of

commercial and cloud systems at scale– Implement a hybrid public/private/multi-tenant cloud

– 1024 public IP addresses – enough to support the largest computing system built so far

• Begin the CTE system integration with high-speed interfaces– All the systems have 1G interfaces, will implement 10G, 40G Ethernet and Fibre

Channel by the end of Q3 FY2014

• Integrate:– All the systems into the central management, provisioning, and configuration

control services

– Machine provisioning with software defined networking (SDN)

• Implement the ability to provision virtual and physical machines with complex applications and databases pre-loaded

• Enable a vastly simplified complex system/architecture implementation-integration-interoperability capability 3

Initial CTE Technology Focus Areas• Establish the central CTE lab infrastructure at the

QTS Richmond facility• Support connectivity and integration with the

satellite facilities• Demonstrate high performance facility integration• Ingest an initial set of technology partners to

explore a variety of cloud computing infrastructure hardware and software

• User focused demonstrations

4

CTE Technology Ingest Process• Technology provider interviews

– Organization and key person assessment– Technology overview, typical use cases, existing or

envisioned customer base– Standards compliance– Open Systems/Open APIs

• Use Case Mapping• Modifications and/or Interoperability Implementation• Component/System Integration• Use Case Test/Demonstration

5

Use-Case Approach to T&E• Isolated technology tests are of limited value• Test both the technology/service and the

providing organization • Provide quantifiable value to participants– Metrics of various types– Reference third party installation and test– Neutral test, integration, and demonstration

capability– New partners and business opportunities

6

GPU-based Investigations

• HP-nVidia Partnership Testing• Video and Imagery Analytics• Computational Astronomy• Computational Cryptography• Computational Neuroscience• Cloud-based Robotics Control and

Management• Immersive Stereoscopic Visualization

8

CTE Prototypes and Use CasesNumber Name Sponsors Short Description

CTE 13-1 DIY Cloud Infrastructure: Rapid Deployment of a Private and Public Cloud with Amazon-style Ease-of-Use and Accounting

HP, Convergence HP Demonstration of the Cloud Matrix for IaaS, Virtual Desktop demo with Convergence user devices

CTE 13-2 Intercontinental, Large-scale VIDINT Command/Analysis Center: Distribution, Storage, Full Field of View Tracking, and full video and image stream search with arbitrary image(s)

Pixia, Aspera, Signal Innovations, pixLogic, Flexanalytics, Cloud Front Group, L3, HP, M3 Com, Ciena, Infinite Dimensions, Terramajic

Show full field of view tracking and analytics demo with metadata transfer to Autonomy advanced analytics

CTE 13-3 Tapping the Firehose: Large-scale Social Media Analytics and Command Center

InTTENSITY, Optensity, Infinite Dimensions Working with data sources such as the main Twitter feed is challenging. A social media command center will be demonstrated, providing insight into a variety of topics.

CTE 13-4 Making Big Data Pay: Visual Programming of Advanced Analytics and Manipulation of Data

Optensity, Flexanalytics, Pixlogic Demonstrate an easy-to-use visual programming tool for advanced analytics. Integrate Flexanalytics/Pixlogic video analytics.

CTE 13-5 Beyond the OODA Loop: Collect-Correlate-Alert-Act (C2A2) Attivio, Ringtail A demonstration of decision data is drawn from multiple sources, correlated, and alerts generated and sent for actions to be taken in the context of shipping port management and security.

CTE 13-6 Multi-INT Enterprise Service "Dial tone" and Collect-Correlate-Alert-Act

MarkLogic, Ringtail A multi-INT scenario illustrating the Collect-Correlate-Alert-Act (C2A2) paradigm.

CTE 13-7 Next-Generation Cloud-based C4ISR Visualization Ringtail A next-generation command center, used by the Chairman of the Joint Chiefs of Staff , in terms of both software and hardware.

CTE 13-8 Big Data Storage and Analytics Oracle, Platfora, MapR, Cloudera, Hortonworks, Sqrrl, IBM, Security First

Data cloud protection, visual tools and analytics for Hadoop-based systems in multiple facilities

CTE 13-9 Object Storage and Analytics Hitachi Large scale object store and manipulationCTE 13-10 iCommand Dell, AAI, Textron Next Generation Command Center using Chairman of the Joint Chiefs of Staff

visualization systemCTE 13-11 Video Management KSI Video Social Media System for Video and Image AnalysisCTE 13-12 Asteroid Detection/Computational Astronomy UN Space Programme II, NASA, ESA The CTE supported an international team in an asteroid detection contest. The

CTE team discovered a previously unknown Main Belt asteroid, a rare achievement.

CTE 14-1 Large-scale Immersive Training and Evaluation Private sector investors, 5 NFL Teams, OSD P&R Integrate government founded software development kits (SDKs), such as the University of Southern California Institute of Creative Technology's Virtual Human Toolkit, and create a next-generation immersive training environment, with improved SDKs.

CTE 14-2 COTS Test and Evaluation NRO and others Formally test COTS technology and provide evaluations to interested government agencies. NRO has started working with the CTE to evaluate the high-speed message acceleration hardware employed by large-scale stock and bond trading organizations.

CTE 14-3 Bitcoin Mining: Computational Cryptography CTE Finding bitcoins, a process refered to as bitcoin mining, is an exercise in computational cryptography. The CTE is strtudying the algrothms and technology associated with bitcoin mining as community innovations in gryptography, and real experience with bitcoins. Bitcoins are used by criminals and others who wish to obscure financial transactions.

CTE 14-4 Automating Identification FBI, Verizon, Infinite Dimensions Efforts such as the FBI's Next Generation Ientification (NGI) program need to utilize a multi-source data to ID a variety of entities and concepts. This project evealuates and integrates technology to provide insight into promising technologies that are relevant to this task.

DIY Cloud Infrastructure

• Show how easy it was to integrate various cloud systems through a single and unified provisioning and cost accounting system

• Provision both virtual and physical machines (for performance)

• Permits a better match between task and execution architecture

• Eliminate vendor lock-in

9

Features• Infrastructure-as-a-Service (IaaS) for private

and hybrid cloud environments• Self-service infrastructure portal for quick auto-

provisioning– Integrated billing

• Support of heterogeneous environments– Cloud-bursting to a variety of public cloud providers– Supported hypervisors include VMware, Microsoft

HyperV, Red Hat KVM, and Integrity VMs

10

Large-Scale VIDINT, IMINT“Netflix for the DoD/IC”

• Used the software transport software used by Netflix to move the video and imagery

• Managed video and imagery using Pixia’s system• Employed Project Guppy’s long-distance, high

performance networking – Aspera’s London lab to QTS Richmond

• Integrated:– Video and imagery analytics on ingest using Signal

Innovation Group’s software to track all moving elements of a video or image stream

– Video and imagery search using pixLogic’s system

• Integrated into HP Autonomy’s correlation and alerting engine 11

Performance Metrics• No observable latency in the transmission– Phase III hypothesis: Project Guppy team can provide

affordable high performance transoceanic circuits that can deliver high volume and velocity content anywhere

• Days for the key integrations:– The Pixia, SIG, and pixLogic integrations were

facilitated via STANAG 4559, NATO Standard Image Library Interface

– Open API’s facilitated the integration into HP’s Autonomy

– Phase III hypothesis: Other video and image analytics, correlation, and alerting engines can be easily integrated 12

Large-Scale Social Media Analytics and Command Center

• Integrates full Twitter Firehose, Facebook, etc.• Data is injected into a real-time ingestion and

orchestration engine that allows creation of distinct processing pipelines to filter

• Performs specific Natural Language Processing (NLP) based enrichment of the social media data of interest

13

Visual Programming of Advanced Analytics – Video Analytics Integration

• Rapidly compose and execute data analytics across multiple clouds

• Graphical composition for analytic workflows, virtualizes the composed workflows (or Apps)

• Executes the Apps across multiple clouds• Integrated with piXlogic– Provides visual search solutions that automatically

analyze and index the contents of images and video files

14

Beyond the OODA LoopCollect-Correlate-Alert-Act (C2A2)

• Integrated Tibco’s Spotfire analytics capability with Ringtail’s Common Operational Picture visualization use by the Chairman of the Joint Chiefs of Staff (CJCS)– Integrated open source data about ship

movement between and in ports to Spotfire– Correlation calculations triggered Alerts in Spotfire – Alert messages from Spotfire sent to the CJCS

Command System

15

Performance Metrics

• Spotfire integrated and collected multiple open source data feeds into the correlation process

• The Spotfire-CJCS COP integration was performed in less than one week

• The Java Message Service (JMS) was leveraged to provide interoperability

• Open APIs also facilitated integration

16

Multi-INT Enterprise Service "Dialtone" and C2A2

• Experimented with the notion of an enterprise service registry

• Implemented via the Representational State Transfer (REST) construct

• Integrated multiple data sources and presented them to an enterprise search function– Both unstructured and structured– Demonstration employed open source government

data

17

Next-Generation Cloud-based C4ISR Visualization

• A more complete demonstration of Ringtail’s Common Operational Picture visualization use by the Chairman of the Joint Chiefs of Staff (CJCS)

• Data and capability integration and interactivity are supported via:– Well-defined JMS-based messaging– REST interface support

• Video and collaboration integration to overall package was interesting

18

Big Data Storage and Analytics

• Showcased Data Cloud Protect participants• Included Hadoop vendors and analytics

partners• Provided a venue to educate the audience on

the operational issues • Highlighted initial collaborative efforts and

results

19

Object Storage and Analytics

• Installed and implemented Hitachi Data Systems prototype Object Store highlighting:– Multiple Entry Points: The system must allow for multiple independent

applications simultaneously performing operations such as read and write.

– Global Namespace: An object store should present a global namespace (GNS) to the client.

– Access Protocol: The access protocol should work equally well over a WAN (such as the Internet) as over a LAN.• Therefore, the access protocol cannot be chatty, as most network file

system protocols tend to be• Further, it should support mobile devices, such as smartphones, as clients

– Unstructured Data: The system must be designed to optimize for unstructured data as this will be the predominant type

– Hardware Agnostic: Hardware changes frequently. This includes servers, storage subsystems, and even the storage medium

20

Social Media and Video Management

• Cloud-based video and associated “Big Data” management services

• Focus is video/data management and sharing from “Remote Sensing Platforms” (e.g., unmanned aircraft, boats, underwater vehicles)

• Social media-like interface and approach to content management, and analytics integration

21

VIDINT Command Center

• COTS-based approach to a Video Intelligence Command Center

• Focused around cameras and facility physical security

• Integrates analytics capable of license plate recognition (LPR) with the camera feeds – Employs commercial video formats and

interchange standards

22

Summary

• Covered quite a variety of systems, both hardware and software

• Managed to establish many noteworthy performance metrics for partners to tout

• Illustrated the value and use of standards, open systems, and agile development methods

• Set the stage for Phase III and expanded integration of the CTE infrastructure

23

Background

Continuous Transformation Environment

Overview and Current Direction

“A Cloud Computing Range”

• CTE Objectives– Prototype development and experimentation: Innovation in a

collaborative environment by industry, government, and academia in an open systems collaborative environment

– Integration, verification, test, and release of Commercial-Off-The-Shelf (COTS) products for government use

– Rapidly prove operational utility of high technology solutions– Open systems and standards compliance evaluation, documentation,

and capabilities matrix

• Solve GAO identified big integrator problem1

Organizing principle: “Give innovation a chance.”• Consortium of large and small technology companies

and facilities partners

CTE at a Glance

261 Government Accounting Office-09-326SP, http://www.gao.gov/new.items/d09326sp.pdf

CTE Network

27

Quality Technology Services (QTS) and Verizon (VZW) are the current facilities providers OCONUS Sites are VZW CONUS VZW: Engelwood, CO; Culpeper, VA, Miami, FL Remaining CONUS Sites are QTS, in particular the 1.3M sq. ft. Richmond, VA site

Main CTE Experimentation Lab LocationRichmond- High Density Multi-Data Center Campus

500,000 Sq Ft of Planned Raised Floor Multiple Distinct Data Center Buildings

(Current Basis of Design)

1

3

2 Office Space

1.3 Million Sq Ft Campus

Use Case Scenarios1. Industry product for government use2. Government product for industry use3. Industry/government collaboration for government

use4. Standards Evaluation for government and/or

standards body5. Industry Compliance and Confidence testing6. Government interoperability testing of industry

products to select replacements or build partnerships7. Government operational evaluations/ttp

development of cloud solutions8. Discovery and reuse through collaboration of

communities of interest

Use Case Scenario Development

• Find Opportunities from Suppliers and Users

• Develop and Evaluate Opportunities

• Supply Confidence/Knowledge to Consumer

• Field and Use

Government Opportunities• Resource seed prototypes, enabling rapid industry

driven cloud solutions to DoD problems

• Establish DoD CRaDA with the CTE that enables mutual benefit in:

– Open System, Standards, and Content

– Compliance testing results/methods

– Open system industry demonstrations for DoD consideration

– Open Government data/systems for industry consideration

• Establish Open Source/Open Systems Charter foundation for DoD, including repository functions

31

Mutual Benefit

Industry/CTE Gain• Direct Rewards to

Innovators• Provide the Opportunity for

Agile Partnerships• Access Government

Solutions/Information (e.g. GOTS Software)

• Improve Organizational Focus and Strategy Execution

• Support Ongoing Innovation

Government Gain• Implement Agile

Development and Contracting Strategies

• Speed New Applications and Systems from Concept to Use

• Enhance Efficiency in Fielding and Support

• Outpace Emerging Threats• Optimize Resource

Employment

Way Forward/Next Steps

• Invite Government Leaders back to CTE for workshop

– November (OSD, Joint Staff)

• Develop Proposal for Prototypes – December (CTE)

• Develop Draft CRaDA – December (OSD, CTE)

• Develop White Paper on Open System/Open Source

Function – January (CTE, OSD)

• Seed into field activities (AF ACC, AF/Joint Training,

TRMC S&T) – May (CTE, OSD) 33

34

Summary

• Continue to evolve the capacity and capability of

the CTE with industry partners

• Form government partnership to create a gateway

for government-industry collaboration

• Establish a series of rapid prototypes for DoD and

non-DoD solutions

• Demonstrate open systems/open source expertise

CTE Background

35

• QTS• HP • Hitachi Data Systems• Verizon• Oracle• NetApp• Solace• Ciena• IBM• ISC8• Quantum• Spectralogic• Blackridge• EMC• Nicera• Juniper

36

CTE Infrastructure

Access to over: 50,000+ Processing Cores

100,000 reserve 100+ Petabytes of Storage Over 30 Sites Worldwide

Classified/Unclassified FacilitiesAnd Growing…

Main facility in Richmond, VA: 1.3M square feet 110MW of power from 2 grid

elements Full onsite backup & UPS

25,000 tons of chilled water generation capacity per day

CTE Software Participants• Cloud Infrastructure

– Amazon, Dell, EMC, HP, IBM, Oracle, Quantum, Red Hat, Aspera

• Data Storage and Manipulation– Accumulo, Cloudera, Hortonworks, MapR, Sqrrl, Vertica/HP

• Document Manipulation and Analytics– HyLighter, IxReveal, Digital Reasoning

• Video Management and Analytics– Pixia, SIG, Aspera, Pixlogic, Flexanalytics, Terramajic, KSI Video

• Advanced Analytics Tools– AgileX, HP, Attivio, IBM, Optensity, Oracle, Solace

• Security– Blackridge, CyberPoint, IBM, ISC8, Red Hat, Security First, Verizon

37

Current Technical Emphasis• Complete the CTE System Integration with High-Speed

Interfaces– All the systems have 1G interfaces, will implement 10G by the

end of Q1 FY14

• Integrate all the systems into the central provisioning service

• Implement the ability to provision virtual and physical machines with complex applications and databases pre-loaded

• Integrate machine provisioning with software defined networking (SDN)

• Enable a vastly simplified complex system/architecture implementation capability 38