Upload
willis-kennedy
View
223
Download
0
Tags:
Embed Size (px)
Citation preview
MIIC-2 Kick-Off MeetingMay 28, 2014
Pamela RinslandMike Little
ATMOSPHERICSCIENCEDATA CENTER
04/19/232
ATMOSPHERICSCIENCEDATA CENTER
AGENDA
• ASDC Web Services• MIIC-2 Services Architecture• Support Plans and Issues
ASDC Areas of ModernizationFocus on Discovery and Access
Data Stewardship Data Distribution Data Production
ASDC
Cloud ComputingGIS OPeNDAPiRODS
• Partners: AWS, OCIO• Planning: 2013 - present• Experiments: June ‘14• IT Lab “cloudbursting”
(ARC, GSFC)
• Partners: NASA NCCS, RENSI, HIFLD, OPeNDAP• OPenDAP/Giovanni operational; MIIC-1 prototype• GIS experiments close to Operational Status• Parameter selection based on Ontology - ODISSEES• iRODS in progress
Ontology
SemanticTechnology
Service Oriented
Architecture
DigitalLibrary
Items in green are areas in work; Items in yellow are in still in preliminary conceptual planning phase
AnalyticTool
Support
Hadoop
04/19/234
ATMOSPHERICSCIENCEDATA CENTER ArcGIS Prototype with CERES EBAF Surface Data
GIS Use Case CERES Surface Flux with Freshwater Lakes
04/19/236
ATMOSPHERICSCIENCEDATA CENTER MIIC-2 Services Architecture (Functional View)
04/19/237
ATMOSPHERICSCIENCEDATA CENTER MIIC-2 Services Architecture (Hardware View)
04/19/238
• ASDC– Characterize system at ASDC “as is” to determine
bottlenecks and saturation (what if we have 10 calibration plans executing at once?) so we can deploy a responsive MIIC-2 configuration
– Upgrade to NFS over 10GigE– Re-characterize the upgraded system using
AppDynamics to monitor resource consumption– Refine architecture to reflect path to scalability– Identify/acquire MIIC2 services server and MIIC2
services PostgreSQL DB server– Acquire a Load balancer to handle CPU loads on MIIC2
services server(s); i/o loads on OPeNDAP servers
ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues
04/19/239
• NCDC– Determine available/need for a MIIC-2 deployment
“box-wise” ($15K can get a decent server)– Determine latency drivers: off-campus pipe width
(10Gbps? OC-48? etc.) Internal networking– Confirm target files at NCDC to publish via MIIC-2;
Confirm files at NCDC in a format that can be handled through standard OPeNDAP? (HDF5?)
– Determine what NCDC staff expect to be able to install and verify the MIIC-2 services (Delivery package)
ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues
04/19/2310
• NASA-LaRC and NCDC - Space Act Agreement
• Determine approach: Umbrella with annex or “one and done” (current draft example)
• ESDIS/ECHO a separate effort– Conversation with Jeff Walter last week (Andy Mitchell was
unavailable) but need to close the loop
• Signature levels required• Use NASA Space Act Agreement Maker (SAAM) tool• Rudimentary milestones required
– MIIC-2 schedule is one candidate, but don’t want to add more bureaucracy
ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues
04/19/2311
• NASA-LaRC and NCDC – Schedule of Tasks and Performers
• Determine schedule for deliverables and activities to support ACCESS proposal 2-year period of performance
• Technical POCs for tasks• Communication approach within team (internal and
ahead of required ACCESS reporting)
ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues
04/19/2312
• NASA-LaRC and NCDC – work through support plan items in later sessions
• Schedule of Tasks and Performers• Communication approach within team (internal and
ahead of required ACCESS reporting)• SAA Approach
ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues
04/19/2313
ATMOSPHERICSCIENCEDATA CENTER
MIIC-2 Project Wiki Status Pages – Ready for Members to Join in!
04/19/2314
ATMOSPHERICSCIENCEDATA CENTER
Background
04/19/2315
• NFS path over 1 GigE for now from MIIC-2 through OPeNDAP to ASDC Data holdings on the 1 PB DPO (CALIPSO and CERES, captive 1 month each of VIIRS and GOES data)
• Hyrax 1.9 with MIIC-2 plugins is installed on datasrvr2 (latest and greatest version from the OpENDAP group for custom MIIC services) datasrvr2 is an IBM x3650M2 with two Intel Xeon E5-2650 CPUs. The CPUs are 8-core 16 thread CPUs running at 2.0Ghz, it has 32GB of memory. It is running Redhat Enterprise Linux 6.3, it currently has a dual port 10g card installed (not configured, but has 2 10GigE GBICs), and 4 1GigE ethernet ports, three of which are configured. The server is located on ROW I, RACK D, at rack position 26.
• MIIC-2 services in Tomcat container/Apache with UI web page that comes up on client browser and also RESTful API for m2m batch – currently on desktop in Eclipse environment! PostgreSQL DB is on desktop so actual performance is skewed until more realistic configuration is achieved. Things are moving on to datasrvr2.
ATMOSPHERICSCIENCEDATA CENTER ASDC MIIC-1 Configuration
04/19/2316
• Development Environment: Aron installing an Eclipse Metrics plugin and will present its findings. A bug in Metrics necessitates a temporary downgrade to Java 1.6.
• CM tools for now (svn and TRAC)and planned (repository will be Atlassian stash/git)
• Eclipse + Spring tool suite. Migrating to Atlassian Project Mgtmt suite: Confluence, Jira, Bamboo, Stash (ASDC uses Atlassian)
• Testing: Some JUnit tests for unit test and higher-level end-to-end integration test. Currently do not have any coverage type tests.(use Bamboo)
• For the OPeNDAP plugin, use valgrind + memcheck to test for C++ memory leaks. (C++ is used for the OPeNDAP plug-in modules)
• Performance Testing: None currently on the Java side, as our current performance testing to date has identified the biggest performance blocks at the server. On OPeNDAP we use valgrind + massif to quantify memory allocations. (look into AppDynamics)
ATMOSPHERICSCIENCEDATA CENTER ASDC MIIC-1 MIIC-2 S/W Environment
04/19/2317
• TLE provider: Two-Line Ephemeris data are currently accessed from angler and incurs 35-second latency. Need to move information into PostgreSQL DB within MIIC-2 environment and connect to Celestrack (sp?) to get daily updates.
• Database: PostgreSQL, schema is auto-generated by Hibernate annotated Java. Base size is currently trivial (7026k). Need DB size of a one month study. ASDC uses Irwin for schemas.
ATMOSPHERICSCIENCEDATA CENTER ASDC MIIC-1 MIIC-2 S/W Environment
04/19/2318
• Just a template!!! – Demonstrate that ….
• Functional and V&V Testing completed• Infrastructure reviews completed
– Obtain NASA ASDC Management approval to proceed with running ACS-CALIPSO
• Actual schedule for production run to be determined
ATMOSPHERICSCIENCEDATA CENTER Software Packages Status