1NODC, Russia

SeaDataNet TTG meeting

Paris, 18-19 May

Overview and potential use of E2EDM technology for

SeaDataNet

Sergey Belov, Nick Mikhailov

2

Scope

NODC, Russia

E2EDM provides following functionality:

• Wrapping local data systems• Mapping local codes and dictionaries• Data discovery, delivery and processing (export and data

production from delivered data)• Integrated and secured online access to the distributed data

E2EDM is integrating operational and delay-mode time scale data (oceanographic and marine meteorological data) from multiple data formats and multiple data providers in different geographic regions.

3NODC, Russia

Basis

NODC, Russia

• Concept model • Namespace • Metadata/data objects specifications• Data exchange protocol• Software components

4NODC, Russia

E2EDM concept model

NODC, Russia

Providing formalization, identification, description and interpretation for main model objects:

• Data source• Resource• User (in particular cases – external application)Model is based on the following structured semantic objects:• Element• Class - set of elements• Record - set of classesThese semantic objects defines:• Granularity and description for original and transferred data• Metadata description• Codes and dictionaries unification• Data and user policy

5NODC, RussiaNODC, Russia

E2EDM namespace

NODC, Russia

Defines metadata and data structures in terms of XML scheme. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic).

• Global namespace – structures for the model objects• Conceptual (thematic domain) namespace – list of thematic

elements dealing with grouping, e.g. some discipline and available for discovery and exchange

Both designed in XML Schema.1. Global XML Schema (ISO 19115 + WMO Core Metadata +

CDI)2. Concept XML Schema (DiGIR)XML Schemas are also used for validation and development

issues (XML Schema -> UML -> software design)

6NODC, RussiaNODC, Russia

E2EDM metadata objects specifications

NODC, Russia

More than 100 elements, 27 classes, 7 metadata records and 4 data records

Metadata

Information about local data source (linkage)

Unique identification of metadata object

Information about data set

Information about references and responsible party

Information about temporal, vertical extent

Information about data policy

Information about data quality

E2EObjectID

E2ELinkage E2EDataQuality

E2EDistributionInfo

E2EVerticalExtent E2ETemporalExtent

E2EObjectContact

E2EObjectReference

E2ERestrictions List of classes, definingInstruments and processing tools, projects, platforms, geographic coverage, data structure and granularity

7NODC, RussiaNODC, Russia

E2EDM data objects specifications

NODC, Russia

E2EDM data records, based on NetCDF:• E2EDataPoint – for point data• E2EDataProfile – for profile data• E2EDataGrid – for grids• E2EDataObject – object files (http-based and local)

Header (DataHeader) ListElement (list of elements)

DataContainer

«SequenceElement»

«StructureElement»

Common transport NetCDF construction

8NODC, RussiaNODC, Russia

E2EDM data objects specifications (II)

NODC, Russia

class/element NetCDF component NetCDF structure

E2EObjectHierarchy + objectHierarchyLevel

DataHeader + DataContainer

Group

objectHierarchyId DataHeader Attribute

objectHierarchyList DataContainer Variable

Classes and data constructions relationship:

<E2EObjectHierarchy><objectHierarchyLevel>1</objectHierarchyLevel><objectHierarchyList>objectSystemId, objectHierarchyId, countryId, organizationId,

dateTime, Latitude, Longitude, platfType, platformId, platformName</objectHierarchyList><objectHierarchyId>platformId</objectHierarchyId>

<instanceId>platformId</instanceId></E2EObjectHierarchy><E2EObjectHierarchy>

<objectHierarchyLevel>2</objectHierarchyLevel><objectHierarchyList>DEPH, TEMP, PSAL</objectHierarchyList><objectHierarchyId>dateTime, Latitude, Longitude</objectHierarchyId>

<instanceId>dateTime</instanceId></E2EObjectHierarchy>

9NODC, RussiaNODC, RussiaNODC, Russia

E2EDM data exchange protocol

NODC, Russia

Data exchange protocol is responsible for data requesting and data retrieving.

Request message

(XML)

Response message

(XML)

Transport data file

(NetCDF,v.2.2)

10NODC, Russia

E2EDM software components

1. E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files.

2. E2EDM Integration Server – set of services, providing security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data)

11NODC, Russia

E2EDM ArchitectureNamespace

Global XML Schema, conceptual XML

Schema USER

Integration Server Management Service

Metadata service

Data access service

Interface service

Integration Server

Transport service

Request message to data

provider

Response message from data provider

Data Provider

Data Provider Management Service

Database Access Service (DiGIR-based)

Local database

Local database

SQL query

Result

Extended transform

API

Configuration metadata

Data format service

File Access Service (OPeNDAP based)

Local data system(files)

Transport data files

Navigation service

Request message to data

provider N

Response message from data provider N

Response from data provider

Integration Server layer

Communication layer

Data Provider layer

Local data system layer

External layer

12NODC, Russia

Integration Server. Features

• Metadata update & maintenance (automatic)• Resource catalogue management• Discovery services for data, metadata and common codes• Request/response communication service• Data sources monitoring• Downloading services (for obtaining data files, metadata, resource and

instance descriptions synchronization)• Export tools (e.g. export in XML from E2EDM resource description into

WMO Core Profile, to ASCII)• HTTP GET/POST (REST) and Web service interfaces• Security service• Schedule mechanisms• Administration tools (CRM, CMS)

13NODC, Russia

Data Provider. Features

• Data source/resources registry and metadata update• Codes and dictionaries mapping• Request handling• Access to the local data source (DBMS, files)• Data extraction based on search criteria• Transport data file generation• HTTP GET/POST (REST) and Web service interfaces• Security service• Schedule mechanisms• Administration tools (online maintenance for resource and instances)

14NODC, Russia

General data exchange scheme

User

Integration Server

Interface layer (HTTP, Web-services)request

Data ProviderData Provider

Data Provider

Request message

reply

Transport data file

Transport data file

Response message

15NODC, Russia

Current operational status of E2EDM

• E2EDM was originated as one of the JCOMM/ETDMP pilot projects in 2003

MetOffice, UK

IFREMER, France

VLIZ, Belgium

NODC, Russia

16NODC, Russia

Pilot project data sources

The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data

Geographic area – North Atlantic (test-area)

17NODC, Russia

Pilot project data sources

The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data

Geographic area – North Atlantic (test-area)

18NODC, Russia

User data access interface – via Integration Server

Resource catalogue

Parameters

Additional search options

19NODC, Russia

User Data Access interface (cont.)

Search status

20NODC, Russia

• E2EDM Web site http://data.meteo.ru/e2edm/

• E2EDM Integration Server - IODE Project Office (VLIZ)

http://e2edm.vliz.be - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/

User data access interface (cont.)

Point data grid dataprofile data

21NODC, Russia

E2EDM using for SeaDataNet

E2EDM Integration Server

E2EDM Data Provider

Data providers community

HTTP communication

Local Database system Local Data file systemSDN Data Centres

Network

Web-portal

NetCDF transport data-file

Object transport data-file

Responce message

Request message

User access

Registry DiscoveryDelivery

ConnectionEncoding Access

Metadata@Data

22NODC, Russia

E2EDM adaptation for SeaDataNet

JRA2 - Technical development of the interoperable system:E2EDM covers many of JRA2 tasks, but all actions & decisions

should be discussed. E2EDM model deepness is enough for many of the SeaDataNet positions. ( + - there is solution in E2EDM)

• Discovery servicesMetadata update & maintenance +Potential decision: CDI adoption.• Security servicesData access level +Transferred data size level +• Delivery servicesDownloading services + • Viewing servicesDeveloped user interface, libraries, external tools (based on

NetCDF) +

23NODC, Russia

Thank you!

Questions?