Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
D2D TestbedD2D TestbedD2D Testbed D2D Testbed
Sanjeev Mohindra and Julie MullenSanjeev Mohindra and Julie MullenMIT Lincoln Laboratory
DISTRIBUTION STATEMENT A. Approved for public release; distributed is unlimited
This work is sponsored by the Department of the Air Force under Air Force contract FA8721-05-C-0002. Opinions, interpretations,
Testbed - 1SM 3/11/2011
MIT Lincoln Laboratoryconclusions and recommendations are those of the author and are not necessarily endorsed by the United States Government.
Outline
• Introduction• Testbed Overview• Testbed Architecture• Initial Testbed Rollout• Testbed Access• Summary
MIT Lincoln LaboratoryTestbed - 2
SM 3/11/2011
Data to Decision Testbed
D2D testbed is a government-owned prototype of the Data to Decision platform
D2D testbed is a government-owned prototype of the Data to Decision platform
MIT Lincoln LaboratoryTestbed - 3
SM 3/11/2011
the Data to Decision platform.the Data to Decision platform.
Data to Decision Platform
Data, Apps, Software Infrastructure Distribution
CommunityHardwareD2D Platform
InfrastructureUtility Computing
High Performance Computing
D2D Testbed hosts data and applications and
MIT Lincoln LaboratoryTestbed - 4
SM 3/11/2011
D2D Testbed hosts data and applications and provides storage and compute resources
Scalable InfrastructureG id
MultipleParallel File
LLGrid
Single Disk
Multiple Disks System
Storage
Cluster
GridCloud
g
Single Machine
Cluster
Computation
Cluster(s) TX-X TX-DoD
Classification External RestrictedCompute Nodes 120 35TFLOPs 3 1 0 9TFLOPs 3.1 0.9Central Storage (TB) 14 8Distributed Storage (TB) 9 12
MIT Lincoln LaboratoryTestbed - 5
SM 3/11/2011
Infrastructure supports storage and computation on large data
Outline
• Introduction• Testbed Overview• Testbed Architecture• Initial Testbed Rollout• Testbed Access• Summary
MIT Lincoln LaboratoryTestbed - 6
SM 3/11/2011
D2D Testbed Layer Diagram
Analytic Layer
User Interaction LayerLibraries of UI modules
Data Management Layer
Analytic LayerLibraries of analytic modules
Data Management Layer• Service-Oriented Architecture• High Performance• Scalable
RKA D2D Presentation
Testbed users should be able to access data; buildTestbed users should be able to access data; build and consume analytic services; and interact with data and services.
MIT Lincoln LaboratoryTestbed - 7
SM 3/11/2011
Testbed Management
Development Testbed
Mature and Tested Apps can be transitioned into the D2D “Production” Testbed
Mature and Tested Apps can be transitioned into the D2D “Production” Testbed
Test on “restricted data”
Test on “restricted data”
(Open) Production TestbedProduction Testbed
Development Testbed
•Selected non-sensitive data
•Open Apps•Data and App
(Restricted)
D2DData and App Interfaces
•Collaboration tools•Selected data•Apps•Data and App interfaces
“Production” Testbed
•Collaboration tools•All data•Selected apps•Data and App interfacesC ll b ti t lConsortium drives continuous
MIT Lincoln LaboratoryTestbed - 8
SM 3/11/2011
•Collaboration toolsConsortium drives continuous improvement and transition
Outline
• Introduction• Testbed Overview• Testbed Architecture• Initial Testbed Rollout• Testbed Access• Summary
MIT Lincoln LaboratoryTestbed - 9
SM 3/11/2011
Types of Testbed Users
• Develops analytics in Matlab• Knows that SOA exists
Analytics DeveloperAnalytics
DeveloperInitialFocusInitialFocus
• Programs in Java• Loves SOA
Software DeveloperSoftware
Developer
S tS t • Hosts testbed• Makes apps work together
System IntegratorSystem
Integrator
• Analyzes data to make decisionsA l tA l t • Analyzes data to make decisions using apps AnalystAnalyst
Architecture should keep the needs of various users in mind
MIT Lincoln LaboratoryTestbed - 10SM 3/11/2011
Data as a service
• Users can mount data on their client machines• Apps can access data as if it were localApps can access data as if it were local
U t d t th h l l t i• Users can request data through local or remote service apps• Apps can access through other data service apps• Data, and data services can be at different locations
MIT Lincoln LaboratoryTestbed - 11SM 3/11/2011
Provides users with access to data hosted on testbed
Analytics as a serviceFile Services
Storage
Data Server
Data ServicesAnalytic Services
Web Service
UI
DataData
Server
Database Server
Service
Web Service
• Analytic services interact with data directly or through data iservices
• Client user interaction apps interact with analytic apps• Users can run analytic apps locally or remotelyy pp y y• Running analytic apps on the testbed provides data locality
and scalability
MIT Lincoln LaboratoryTestbed - 12SM 3/11/2011
Users can build and consume analytic services
Matlab Analytic ServerFile Services
Storage
Data
Data ServicesAnalytic Services1
DataData
Server
Database Server
2
Web Service
MatlabAnalytic Server
• Analytic Developer builds apps in Matlab
Server
• To convert to analytic services:1. Write a Matlab calling function and simply place the Matlab
code at a known location2. The Matlab calling function is automatically exposed as a web
service, that can be called using RESTful web service calls
MIT Lincoln LaboratoryTestbed - 13SM 3/11/2011
Analytic developer can focus on algorithms
Outline
• Introduction• Testbed Overview• Testbed Architecture• Initial Testbed Rollout• Testbed Access• Summary
MIT Lincoln LaboratoryTestbed - 14SM 3/11/2011
Initial Testbed Rollout
File ServicesStorage Data Services§
•Image serverSecure Connection
CLIENT MACHINE TESTBEDFor Analytic Developer
DataData Data§• Bluegrass data• Other data
Mount Data remotely
Collaboration Version Control
A l ti A §
Data Services§• Image server
Cluster(s) TX-X TX-DoD
Classification External RestrictedCompute Nodes 120 35TFLOPs 3 1 0 9
Analytic Apps§• Region tracker• Entity tracker
API TFLOPs 3.1 0.9Central Storage (TB) 14 8Distributed Storage (TB) 9 12
API• Region tracker• Entity tracker• Image server
MIT Lincoln LaboratoryTestbed - 15SM 3/11/2011
§Some data and apps only made available on the restricted testbed
Recommended Practices
DataData
Step 1• Mount data• Develop applications on local machinep
Step 2• Build and test services on local machine
Step 2
Step 3• Parallelize applications• Test and deploy on testbedStep 3 Test and deploy on testbed
MIT Lincoln LaboratoryTestbed - 16SM 3/11/2011
Don’t let these get in the way of progress
Outline
• Introduction• Testbed Overview• Testbed Architecture• Initial Testbed Rollout• Testbed Access• Summaryy
MIT Lincoln LaboratoryTestbed - 17SM 3/11/2011
Initial Testbed Rollout
Secure Connection
CLIENTS For Analytic Developer
Mount Data remotely
• Requesting an Account• Accessing the Testbed
Cluster(s) University Government
Classification External RestrictedCompute Nodes 120 35TFLOPs 3.1 0.9Central Storage (TB) 14 8
• Use Policies
Central Storage (TB) 14 8Distributed Storage (TB) 9 12
MIT Lincoln LaboratoryTestbed - 18SM 3/11/2011
Requesting Access to the D2D Collaborative Environment
P d t• Proposed user must – Complete Account Request Form which asks for:
User’s NameUser’s Status (e.g. Consultant, Subcontractor, Student)User’s CitizenshipUser’s company or organizationP i d f ti th t t d t b tiPeriod of time that account needs to be active
– Not share the account with colleagues or students – Restrict their cluster use to program related work
T tb d t ill id• Testbed team will provide– Datasets– Image and Analytical Web Service Interfaces– User Guides for
Accessing and using the Testbed clusterUsing the Testbed infrastructure to develop services
MIT Lincoln LaboratoryTestbed - 19SM 3/11/2011
Accessing the Testbed
Connect via ssh Requires:Requires:
•User Name•Pin•SecureID
XXXXXX
MIT Lincoln LaboratoryTestbed - 20SM 3/11/2011
Mount the Testbed File System in Your Local Workspace
XXXXXX
XXXXXXXXXXXX
XXXXXXXXXX
Mount Testbed File system on local machine
Drag & drop files
Run services
MIT Lincoln LaboratoryTestbed - 21SM 3/11/2011
Outline
• Introduction• Testbed Overview• Testbed Architecture• Initial Testbed Rollout• Testbed Access• Summary
MIT Lincoln LaboratoryTestbed - 22SM 3/11/2011
Summary
• Testbed architecture provides a way for users to access data; build, consume and host analytic ; , yservices; and interact with data and services.
• Analytics developer can focus on developing algorithms, and leave the SOA aspects to the testbed middleware
• The final testbed architecture can be hosted at any• The final testbed architecture can be hosted at any government team member site.
• Testbed will be online before the performers areTestbed will be online before the performers are selected
MIT Lincoln LaboratoryTestbed - 23SM 3/11/2011