Ocean Observatories InitiativeOcean Observatories Initiative

OOI CyberinfrastructureData Management

Michael Meisinger & David Stuebe

OOI Cyberinfrastructure Life Cycle ObjectivesMilestone Review, Release 1

San Diego, CAFebruary 23-25, 2010

OOI CI LCO Review, Feb 20102

Overview

Michael Meisinger 

• Architectural Context

David Stuebe

• Functional Scope in R1

• DM Architecture

• Design and Feasibility

• OOICI Data Model

Ocean Observatories InitiativeOcean Observatories Initiative

Data Management Architecture Overview

OOI CI LCO Review, Feb 20104

Overview

• The Role of DM in the System

• Scope in Release 1

• Common Data and Metadata Model

• Core Services

• Example Scenario

OOI CI LCO Review, Feb 20105

The Role of DM in the System

• Wherever information is required in the system (e.g., science data, metadata, associations, user identities, virtual machine images)– Provide access in the form needed– Distribute– Transform– such that no data is ever lost and is available where

need and when needed

• DM is core infrastructure and science supporting application

OOI CI LCO Review, Feb 20106

Subsystem Dependencies

2650-00008 OV2 CI

Sensing & Acquisition

DataManagement

CommonOperating

Infrastructure

CommonExecution

Infrastructure

Invoke Services,Dataset Ingest/Transform/ Present,Dataset Pub/Sub,Repository Service

Process Execution,Capability Container

DataManagement

OOI CI LCO Review, Feb 20107

Scope of Release 1

• Define the initial, extensible common data and metadata model for OOI

• Dynamic data distribution services

• Provide a data catalog and repository

• Basic Persistent Storage

OOI CI LCO Review, Feb 20108

Outlook: DM Releases 2 & 3• Advanced persistent archive services• Search and navigation, i.e., browsing

and discovering datasets by metadataand content

• External data access services: interoperability with external communities and applications

• Aggregation service, i.e. classification, categorization, and general grouping of data into collections

• Attribution and association services, i.e., adding metadata attributes for resources

Out of scope

for release 1

OOI CI LCO Review, Feb 20109

Information Resources

“CI governed” Resource• Standard and user

metadata attributes (in OOI convention)

• References to other resources

• Categories– Information resource– Physical (stateful) resource– Taskable resource

2650-00039 OV7 CI

OOI CI LCO Review, Feb 201010

Data Management Services (Dataset View)

2910-00001 OV2 DM

Ingestion Transformation Presentation

Distribution Inventory

Preservation

OOI CI LCO Review, Feb 201011

Data Management Services (Distribution View)

2910-00002 OV2 DM

Ingestion Transformation Presentation

Distribution

InventoryPreservation

OOI CI LCO Review, Feb 201012

Use Cases

• Directly in support of the data scenario– Define a dataset– Fetch data and metadata from an external data

source– Ingest data and metadata from an external data

source– Ingest data from a real-time data source– Distribute increments of a dataset on a data stream– Access data via a catalog. Query and discover and

retrieve

Ocean Observatories InitiativeOcean Observatories Initiative

Data Management Development Overview

OOI CI LCO Review, Feb 201014

Overview

Michael Meisinger 

• Architecture

David Stuebe

• Functional Scope in R1

• DM Architecture

• Design and Feasibility

• OOICI Data Model

Functional Scope in R1

OOI CI LCO Review, Feb 201015

Matt Howard

OceanService

UMASSD

CoolRoom OOI CIOOI CI

free-extras

DM Architecture

OOI CI LCO Review, Feb 201016

free-extras

OOI CIOOI CI

Functional Scope in R1

User (IOOS)

• Find data in the catalog

• Subscribe to a CI data source

• Notification of new data availability

• Access data using DAP, WMS

• Access data w/streaming protocol

OOI CI LCO Review, Feb 201017

Functional Scope in R1

Scientist (IOOS modeler)

• Add metadata to define new data source

• Publish data from the source

• Set policy for collaboration using that data

OOI CI LCO Review, Feb 201018

Functional Scope in R1

Operator

• Configure user data management policies

• Configure dataset caching

• Configure physical storage strategy– Replication– Backup– Holding period

OOI CI LCO Review, Feb 201019

Functional Scope in R1

Service

• Data store can persist and retrieve data

• Block transfer of data

• Streaming transfer of data

• Present data & data catalog

• Ingest external data

• Transform data

OOI CI LCO Review, Feb 201020

OOI CI LCO Review, Feb 201021

Overview

Michael Meisinger 

• Architecture

David Stuebe

• Functional Scope in R1

• DM Architecture

• Design and Feasibility

• OOICI Data Model

DM Architecture

OOI CI LCO Review, Feb 201022

Matt Howard

OceanService

UMASSD

CoolRoom OOI CIOOI CI

free-extras

DM Architecture

OOI CI LCO Review, Feb 201023

free-extras

OOI CIOOI CI

DM Architecture

OOI CI LCO Review, Feb 201024

S&A P&P A&S

DM

COI CEI

free-extras

DM Architecture

OOI CI LCO Review, Feb 201025

S&A P&P A&S

COI CEI

Ingestion Transformation Presentation

Distribution Inventory

Preservation

free-extras

DM Architecture

OOI CI LCO Review, Feb 201026

Ingestion Transformation Presentation

Distribution Inventory

Preservation

2910-00001 OV2 DM Services

DM Architecture

OOI CI LCO Review, Feb 201027

Ingestion Transformation Presentation

Distribution

InventoryPreservation

2910-00002 OV2 DM Services

DM Architecture

OOI CI LCO Review, Feb 201028

Distribution UI

Publish Registration

SubscriptionRegistration

NotificationRegistration

Data Stream Registry

COI: ResourceRegistry

Data Stream Routing

COI:Exchange

2910-00014 OV2 DM DistributionDM: Inventory

DM Architecture

OOI CI LCO Review, Feb 201029

2910-00020 OV5 DM Register …

DM Architecture

OOI CI LCO Review, Feb 201030

2910-00022 OV6 DM Registration

DM Architecture

OOI CI LCO Review, Feb 201031

S&A P&P A&S

DM

COI CEI

free-extras

OOI CI LCO Review, Feb 201032

Overview

Michael Meisinger 

• Architecture

David Stuebe

• Functional Scope in R1

• DM Architecture

• Design and Feasibility

• OOICI Data Model

Design and Feasibility

OOI CI LCO Review, Feb 201033

LDM

Application Level Technologies to Leverage

pydap

Transformation & Presentation

Ingestion

Java

CF Metadata

Design and Feasibility

OOI CI LCO Review, Feb 201034

Infrastructure Level Technologies to Leverage

Proto Buffers

Inventory & Resource Registry

Message Encodings

Development Language

Transport

Design and Feasibility

OOI CI LCO Review, Feb 201035

Preservation Technologies to Leverage

Storage Resource Management Specification Compliant

Berkeley Storage Manager

Design and Feasibility

OOI CI LCO Review, Feb 201036

Presentation & Discovery Technologies to Leverage

CUAHSI

Design and Feasibility

OOI CI LCO Review, Feb 201037

Presentation & DiscoveryTechnologies to Leverage

OGCSEARCH

OOI CI LCO Review, Feb 201038

Overview

Michael Meisinger 

• Architecture

David Stuebe

• Functional Scope in R1

• DM Architecture

• Design and Feasibility

• OOICI Data Model

OOICI Data Model

OOI CI LCO Review, Feb 201039

We have to get this right.

OOICI Data Model

OOI CI LCO Review, Feb 201040

Common Data Model

DAP 2 HDF 5

OOICI Data Model

OOI CI LCO Review, Feb 201041

ISO 19123Coverage Types

OOICI Data Model

OOI CI LCO Review, Feb 201042

The existing data models have a physical data dependence, an artificial coupling between algorithm and representation. The GridFields model, separates the scientific question from how to solve it in order to

improve interoperability and performance.

Data Management

OOI CI LCO Review, Feb 201043

S&A P&P A&S

DM

COI CEI

free-extras

OOI CI LCO Review, Feb 201044

Thanks !