Ocean Observatories Initiative Common Execution Infrastructure (CEI) Overview Michael Meisinger...

Ocean Observatories Initiative

Common Execution Infrastructure (CEI)

Overview

Michael Meisinger

September 29, 2009

OOI CI Kick-Off Meeting, Sept 9-11, 20092

Outline

• Subsystem Architecture Overview• Scope of Release 1• Selected Components

– Elastic Computing– Global Network Strategy

3OOI CI Kick-Off Meeting

Sept 9-11, 2009

Common Execution Infrastructure

• Provides the execution and deployment environment for CI services and user processes

Common Operating Infrastructure

Data Management

(Science)

Sensing & Acquisition

Data Management (Information Distribution)

Analysis & Synthesis

Identity Management

State Management

GovernanceFramework

Resource Management

Planning & Prosecution

Exchange

Service Framework

Presentation Framework

Common Execution

Infrastructure

Deployment Scenario

4OOI CI Kick-Off Meeting

Sept 9-11, 2009

Scope of Release 1• Virtual Process Execution Management

– Definition of target execution environment independent processes

– Support of multiple cloud execution environments– Generation of target specific executables

(images)• Elastic Computing

– Elastic scheduling, provisioning, execution– Monitoring and management

• Resource Management– Compute Nodes and Target Environments– Process Repository

OOI CI Kick-Off Meeting, Sept 9-11, 20095

Elastic Provisioning Example

6OOI CI Kick-Off Meeting

Sept 9-11, 2009

• Imperative to separate mechanism from (smart, specialized) policy

Cloud Provisioning Architecture

7OOI CI Kick-Off Meeting

Sept 9-11, 2009

CEI Functional Components

Common Execution Infrastructure

Process Execution & Resource Provisioning

Service Agreement Proposal

Service Agreement Proposal

Status

Fault Analysis

Processing Plan

Deployable UnitDeployable

Type

Execution Environment

Adapter

Fault Monitor

ProvisionerComputation

SchedulerComputation

Controller

OperationalUnit

Execution Engine

Deployable Unit,Control

Status,Fault

Process Definition,Control

Status,Fault

Deployable Type

Deployable Type Repository

Data Management (Information

Distribution) SN

Exchange

Process Definition Repository

Process Definition

Processing Service Requestor

Processing Plan

Operational Unit

Operational Unit

Process Execution & Resource Provisioning

Fault MonitorProvisioner

Execution Resource

ContextualizerExecution

Monitor & Control

Deployable Unit

Operational Unit Status

Operational Unit Status

Control Commands

Service Agreement Proposal

Registration

Resource & Context

Description

ExecutionContext

Service Agreement Proposal

Computation Controller

Execution Engine

Process Execution & Resource Provisioning

Execution Engine

Processing Plan

Fault MonitorComputation

SchedulerComputation

Controller

Execution Engine Dispatcher

Execution Engine Processor

Execution EngineFault Monitor

Service AgreementProposal,

Process Definition

Process Definition

Processing Status

Processing Status

Fault Analysis

Service Agreement Proposal

Example: Data Exchange

Deployment Concepts

12OOI CI Kick-Off Meeting

Sept 9-11, 2009

Operational Architecture

DeploymentArchitecture

ImplementationArchitecture

Execution Engines

13OOI CI Kick-Off Meeting

Sept 9-11, 2009

Deployment - Execution

14OOI CI Kick-Off Meeting

Sept 9-11, 2009

Cloud Provisioning Basics

15OOI CI Kick-Off Meeting

Sept 9-11, 2009

Cloud Instance Startup

16OOI CI Kick-Off Meeting

Sept 9-11, 2009

Observe–Decide–Act Pattern

17OOI CI Kick-Off Meeting

Sept 9-11, 2009

Network Logical Deployment Model

18OOI CI Kick-Off Meeting

Sept 9-11, 2009

Component Deployment Model

19OOI CI Kick-Off Meeting

Sept 9-11, 2009

Hardware Deployment Model

20OOI CI Kick-Off Meeting

Sept 9-11, 2009

Network Architecture

22OOI CI Kick-Off Meeting

Sept 9-11, 2009

Network Deployment

Academic Global Gigabyte Network

Academic National Gigabit Network

OOI Operational LocationsMicrosoft

AmazonWest

AmazonEast

Woods Hole

Seattle

Washington DC

San Diego

Portland

Corvallis

Management Point

Acquisition Point

Distribution Point

Execution Point

Technology Mapping

• Cloud Execution Sites– AWS/EC2, Science Clouds (Nimbus)– GoGrid, FlexiScale (optional)

• National Grid Resource Interface– Condor for high throughput jobs– Teragrid Gateways

Functional Component Technology TRLGrid Scheduler Condor 9Cloud Computing Environment Amazon EC2 9Cloud Computing Environment Nimbus 8VM Packaging & Distribution Elastic Server (CohesiveFT) 8Virtualization Layer Software Individual virtualization vendors, e.g., Amazon AMI)9

OOI CI Kick-Off Meeting, Sept 9-11, 200927

Thanks !

Cloud Computing Strategy• Requirements

– a secure, scalable, fault-tolerant network and computing infrastructure that spans the OOI marine instrument networks to the national (academic & commercial) computing/data facilities

– a participation model that allows individual research teams to integrate the computing environments at their intuitions into this infrastructure

• Strategy– provide an effective development and operational process that allow the

OOI members and its operational staff to transparently use and effectively manage the OOI specific computing resources as a coherent cloud resource

• Definition– style of computing where ICT capabilities are provided “as a Service”, – allowing users to access technology-enabled services without

knowledge of, expertise with, or control over the technology infrastructure that supports them

Cloud Strategy Benefits• Attributes

– Effective coupling of resource acquisition to user demand– Streamlining the preparation, deployment and management of

observatory applications– Removing the management and ownership of physical

computing resources from the direct concern of the Observatory• Results

– Eliminate centralized capital assets associated with networking storage, and computing as well as associated staffing

– Continual transparent tracking of the technology advancements– Ability to respond to changes in usage demand in minutes and

changes in market costs within weeks

Architectural Concerns• Operational

– Enterprise of Applications, Services and the deployed Modules to support the Enterprise

• Deployment (concern of CEI)

– Secure, Scalable, Self-healing Execution Platform

• Implementation (concern of CEI)

– Configuration framework to compose and condition specific classes deployable execution units (Capability Containers)

• Functional– Decomposition of the system based on functional and

construction concerns

CEI Components• 40 min: Elastic scaling and provisioning

– Load graph: Start story. From there to CPE drawings or DX drawing. System scales according to load.

– Domain models development, deployment, operation architecture OV7 overview, execution engine OV7

– CPE use case• 20 min Network: Global site selection (Matthew)

– Deployment architecture– Detailed network diagrams (Brian)– National backbone infrastructure– GLIF (Tom) for network environment we’re deploying into.– Story: National DIF framework using Solace technology, link to

COI/Messaging. Higher density computing per enclosure, diversity of deployment. Possibility for inner core messaging architecture (very efficient on the backbone); commodity equipment

OOI CI Kick-Off Meeting, Sept 9-11, 200931