OBSERVATIONS & PRÉVISIONS CÔTIÈRES Metadata and Data Management Activities winthin NA 3-7 S. Iona (HCMR), D. Schaap (MARIS), L. Rickards

OBSERVATIONS

& PRÉVISIONS CÔTIÈRES

www.seadatanet.org

Metadata and Data Management Activities winthin NA 3-7

S. Iona (HCMR), D. Schaap (MARIS),

L. Rickards (BODC), F. Nast (BSH)

OBSERVATIONS


2Bologna, 19 September 2008

Outlines

• SeaDataNet Objectives

• Training and Capacity building

• Discovery System

• Maintenance

• Upgrade

• Current Status

• Summary

OBSERVATIONS



SeaDataNet Objectives

• To network existing oceanographic data centres already nationally funded

• To develop an efficient distributed pan-European marine data management Infrastructure (a “unique Virtual Data Centre”)

• To provide on-line access to integrated databases of standardised quality by using adapted communication & information technology

OBSERVATIONS



NA3-Training and Capacity buildingAims:

• to ensure that the data and metadata to be integrated in the system will be formatted, checked for quality and disseminated according to the common protocols developed during the project

• to transfer expertise and to train IT experts of the SeaDataNet data centers in the basics, installation and operation of the SeaDataNet technical components

OBSERVATIONS



NA3-Training and Capacity building

Training Workshops, IOC Rroject Office for IODE,

Oostende, (Belgium):(Training material and presentations available on SDN Extranet)

1. February 12-17 2007:

• focused on data re-formatting, quality control, and processing using ODV software.

• training on project metadata and related web services.

OBSERVATIONS



NA3-Training and Capacity building

Training Workshops, IOC Rroject Office for IODE,


2. June 4-5 2007:

• dedicated on generating XML records with the use of MIKADO tool

OBSERVATIONS



NA3-Training and Capacity buildingTraining Workshops, IOC Rroject Office for IODE,


3. June 16-19 2008:

• use of the new V1 formats, interfaces and maintenance tools (MIKADO, online CMS, Web services, Validation services, Vocabularies)

• data quality control and assessments, using ODV software

• analysis and data presentations, using ODV – DIVA software

OBSERVATIONS



Discovery System

The SeaDataNet Discovery System is an integrated catalogue service, aiming at facilitating marine data searching, location and retrieval.

The Objectives are to:

• Maintain and expand the national metadata-bases.

• Standardize the information using common vocabularies and reference tables.

• Interconnect the national inventories in a common Pan-European directory.

OBSERVATIONS



Discovery System

It is composed of several thematic inventories of different levels:

• European Directory of Marine Environmental Datasets (EDMED)

• Cruise Summary Reports (CSR)

• European Directory of the initial Ocean-observing Systems (EDIOS)

• European Directory of Marine Environmental Research Projects (EDMERP)

• European Directory of Marine Organizations (EDMO)

• Common Data Index (CDI)

OBSERVATIONS



Discovery System Structure

OBSERVATIONS



High Level Directories – EDMED, CSR, EDIOSfor describing collected observation datasets

(by ships, by laboratories, by continuous observing system)


OBSERVATIONS



Common Reference Tables – EDMERP, EDMOhold research projects and organizations metadata common to higher directories


OBSERVATIONS



The Common Data Index (CDI)provides access to the data, information and products by data type and/or any other field

distributed by the TA platforms


OBSERVATIONS



Maintenance

Version 0 – 2006-2007

• Continuation and maintenance of existing Sea-Search system :

the data access needs several different requests to each data centres

and the data sets are delivered in different formats

OBSERVATIONS



Discovery System Upgrate

Version 1 – 2008-2010

• Setup of the integrated online data services to users :

networking of 10 “interoperable” data centres of the Technical Task Team

unique request to the interconnected data centres

and the data sets are delivered with a unique format

Progressive integration of 10 data centres by end of 2008

OBSERVATIONS



Current Status• Frozen directories

• Content upgrade from Version 0 to Version 1 using on-line (like CMS forms) and off-line tools that produce XML ISO 19115 compliant exchanges and developed by the Technical Task Team in the joint research activities JRA1,JRA2:

OBSERVATIONS



Cruise Summary Reports (CSR = former ROSCOPs) are the usual means for reporting on cruises or field experiments at sea. Traditionally, it is the Chief Scientist's obligation to submit a CSR to his/her National Oceanographic Data Centre (NODC) not later than two weeks after the cruise. This provides a first level inventory of measurements and samples collected at sea.

Cruise Summary Reports

OBSERVATIONS



New features in SDN V1

The major differences to previous V0 version are:

CSR Local ID given by Data Centre for future updates -

makes it easier to modify/improve existing reports

Most entities now have defined vocabularies

eg. EDMO for organisation, EDMERP for projects and many more

no more spelling/typing mistakes or deviant interpretations

Mandatory fields

improves quality of report

OBSERVATIONS



Structure of Report

Similar to the former ROSCOP forms the Cruise Summary Report has 4 basic parts:

• General Cruise Information

• Mooring Description

• Sampling/Measurement Description

• Information on Geographical Coverage

OBSERVATIONS



How to submit CSRs?

2 methods for generating CSRs

• Online for individual entries (CMS)

• XML files for bulk submission with the use of MIKADO tool

Both tools can be applied for new entries as well as updates !

OBSERVATIONS



Online Content Management System

Link to CSR

Discovery

http://seadatanet.bsh.de/csr/on_line/V1_index.html

OBSERVATIONS



How to update existing reports?

Online update:

• contact BSH-DOD requesting CSRs to be updated

• BSH-DOD loads requested CSRs to entry database

• E-Mail with list of CSR ref. no. and passwords

• modify CSRs in entry database

• save and submit

OBSERVATIONS



How to update existing reports?

Offline – XML update:

• contact BSH-DOD requesting CSRs to be updated

• BSH-DOD sends requested CSRs as XML V0 files

- 1 file/cruise using BSH CSR ref. no. as file name

- all free text information included

• submit modified CSRs in XML V1 format to BSH-DOD

- use BSH CSR ref. no. as central identifier (CSR ID)

- include local identifier (from NODCs) for future updates

OBSERVATIONS



User Interface - Discovery

Resultshttp://seadatanet.bsh.de/csr/retrieve/V1_index.html

OBSERVATIONS



User Interface - Report

Download in V0 format

OBSERVATIONS



Content Conversion into V1

Status of preliminary conversion for 39839 cruise reports:

•Responsible Laboratories (EDMO codes) ~ 20%

• EDMO codes of Chief Scientists ~ 20%

• Ship/Platform (ICES codes) ~ 60%

• Port of departure/return (C381): ~ 30%

• General Ocean Area (C161+C162): ~ 90%

• Sampling units (common vocab. L181) ~ 20%

• EDMO codes of Principal Investigators ~ 3%

OBSERVATIONS



Continuing operation and population

Heraklion/Triest we got 1071 CSRs

• Online 33 %

• XML 34 %

• ICES 25 %

• Others 8 %

Since Triest we got 1263 CSRs

• Online 52 %

• XML 44 %

• ICES 4 %

• Others 0 %

OBSERVATIONS



Status

Number of entries (in an Oracle DB): 39 839

Characteristics: From 1873-2008, 2105 ships from 48 countries

OBSERVATIONS



CSR ContentsCountry Total CSR submissionsUpgraded to V1 New V1 submissions

Albania 1Algeria 7Belgium 22Bulgaria 41Croatia 75Cyprus 2Denmark 5Estonia 8Faroe Islands 45Finland 15 1France 577Georgia 5 2Germany 261 4Greece 49Iceland 2Ireland 102Israel 9Italy 17Latvia 35Lebanon 7Lithuania 14 3Morocco 1Netherlands 14Norway 117 7Poland 14Portugal 3Romania 24Russian Federation 32 3 1Slovenia 14Spain 145 13 18Sweden 152 6Tunisia 91Turkey 38Ukraine 176United Kingdom 27United States 9

2156 16 42

2156 CSR Entries during SDN Project

(1.4.2006 - 15.09.2008)

V1 in operation since 21. July 08

• Russia and Spain with online upgrades

• 42 already using new V1 from 8 countries,

• Lithuania first after the 3rd SDN workshop in Oostende,

OBSERVATIONS



European Directory of Marine Organizations

• The directory lists the organization profiles of all (1000+) Data Holding Centres, Research Institutes, Monitoring Agencies and Research Vessel operators, that have an active role in one or more of the SeaDataNet Discovery services (EDMED - data sets, EDMERP - research projects, CSR - research cruises, EDIOS - observing stations/ systems, and CDI - index to data).

• Direct crosslinks are provided to their entries in these directories.

• The organization entries are maintained online per country by the SeaDataNet partners.

• new Web service for retrieving EDMO entries in XML

OBSERVATIONS



Online Content Management Systemhttp://seadatanet.maris2.nl/vu_organisations/welcome.asp

EDMO CMS geo-locator via Google maps

OBSERVATIONS



EDMO V1 search and retrievalhttp://seadatanet.maris2.nl/edmo

OBSERVATIONS



EDMOEDMO

No. of EDMO entries

No. of EDMO entries

Albania - PUT 0 1Algeria 1 1Belgium 63 69Bulgaria 14 12Croatia 5 5Cyprus 3 3Denmark 36 31Estonia 5 5Finland 9 10France 197 198Georgia 13 14Germany 58 58Greece 59 57Iceland 3 3Ireland 104 46Israel 3 2Italy (OGS +ENEA) 66 60Latvia 9 6Lebanon 2 2Lithuania 7 6Malta 7 7Morocco 3 3Netherlands 11 22Norway 7 23Poland 39 40Portugal 43 45Romania 9 8Russia (SIO + RIHMI) 94 88Slovenia 1 1Spain 47 68Sweden 29 32Tunesia 1 1Turkey 25 26Ukraine 12 10United Kingdom 136 153Other countries 17TOTAL 1122 1134

Country After 1st year SeaDataNet After 2nd year SeaDataNet

EDMO Contents

• Number of EDMO entries is quite stable

• As part of producing CDI records a lot of new organisations have been added; also existing entries have been altered due to re-organisations, double entries etc.

• Content upgrade from V0 to V1 is finished

OBSERVATIONS



EDMOEDMO

EDMED is a directory of data sets relating to the marine environment. It covers a wide range of disciplines including marine meteorology; physical, chemical and biological oceanography; sedimentology; marine biology and fisheries; environmental quality monitoring; coastal and estuarine studies; marine geology and geophysics etc.

European Directory of Marine Environmental Data

OBSERVATIONS



EDMED Contents

Currently, EDMED describes: over 3500 datasets from 700 data holding centres

OBSERVATIONS



Total No. of Datasets and Data Holding Centres

OBSERVATIONS



Progress

• Content Management System will be launched soon

(Content upgrade will start then)

• Web interface is under development

OBSERVATIONS



European European Directory of Marine Environmental Research Projects

EDMERP is a European directory of research projects relating to the marine environment. It covers a wide range of disciplines including marine meteorology; physical, chemical and biological oceanography; sedimentology; marine biology and fisheries; environmental quality; coastal and estuarine studies; marine geology and geophysics etc.

OBSERVATIONS



EDMERP developments

• double entries have been taken out, because projects are entered and maintained by the country of the coordinator, who can add via EDMO entries all related partners

• capability of creation of sub-accounts for institutes in the NODC’s country, while the NODC safeguards the quality by having the chief editor role before publishing

• new Web service EDMERP entries in XML (export and import).

OBSERVATIONS



EDMERP Retrieval

Browse list

Additional details

OBSERVATIONS



EDMERP – CMS

http://seadatanet.maris2.nl/vu_edmerp/welcome.asp

OBSERVATIONS



Total No. of EDMERPs

Total No. of EDMERPs

Albania 0 2Algeria 3 4Belgium 209 128Bulgaria 32 35Croatia 9 7Cyprus 20 9Denmark 40 41Estonia 11 9Finland 16 14France 93 90Georgia 86 102Germany 18 23Greece 114 117Iceland 16 13Ireland 179 160Israel 12 4Italy (OGS +ENEA) 52 48Latvia 31 29Lebanon 5 12Lithuania 28 21Malta 31 30Morocco 5 4Netherlands 71 67Norway 3 4Poland 42 35Portugal 26 26Romania 18 16Russia (SIO + RIHMI) 84 103Slovenia 0 4Spain 91 79Sweden 30 15Tunesia 0 3Turkey 24 28Ukraine 62 79United Kingdom 46 236Other countriesTOTAL 1507 1600

Country

After 1st year SeaDataNet After 2nd year SeaDataNet

EDMERP Contents

• Number of EDMERP entries reduced during upgrading because of removal of duplicates

OBSERVATIONS



European Directory of Ocean Observing Systems

• EDIOS is the European Directory of the Ocean-observing System, a unique searchable metadatabase of observing systems operating repeatedly, regularly and routinely in European waters.

• It contains metadata on European observing systems such as platforms, repeated ship-borne measurements, buoys, remote imagery, etc.

OBSERVATIONS



Progress

• No new input has been requested over the year

• Awaiting new technological developments (XML schema and new version of Mikado)

• BODC has produced an improved Oracle (database) schema

• Supported by common vocabularies, EDMO and EDMERP

OBSERVATIONS



New User Interface: http://seadatanet.maris2.nl/v_edios/search.asp

OBSERVATIONS



Common Data Index

The CDI provides an index (metadatabase) to individual data sets. For comparison: the present European Directory of Marine Environmental Datasets (EDMED) gives an overview of datasets at a high metalevel. Each EDMED data set description covers a broad set of individual measurement data. The CDI gives references to these individual measurement data, providing a more detailed insight into the available datasets.

OBSERVATIONS



New interface

OBSERVATIONS



CDI Contents

Country After 1st year SeaDataNet

After 2nd year SeaDataNet

No. of CDI

entries No. of CDI

entries

Albania - PUT 0 0 See comments Algeria – ISMAL 0 17 Belgium – MUMM 21 21 See comments Belgium – VLIZ 0 912 Bulgaria – IO-BAS 0 606 Croatia – IORS 0 7111 Cyprus – CYODC 0 1056 Denmark – NERI 13676 17042 Estonia – MSI 0 0 See comments Finland – FIMR 0 1667 France – IFREMER 71314 74762 France - CLS 0 8 Georgia – TSU 0 99 Germany – BSH 1594 29295 Greece – NCMR 10588 11588 Iceland – MRI 0 0 See comments Ireland – MI 0 4727 Israel – IOLR 0 4817 Italy - OGS 37926 37926 Italy – ENEA 0 5395 Latvia – IAE-UL 0 104 Lebanon – NCRS 0 7 Lithuania – CMR 0 810 Malta – IOI-MOC 0 0 See comments Morocco – INRH 0 0 See comments Netherlands - NODC 5270 41588 Norway – IMR 13546 14099 Poland – IMGW 0 0 See comments Portugal – IHPT 0 0 See comments Portugal - IST 0 396 Romania – NIMRD 0 2461 Russia – RIHMI 2668 12587 Slovenia – MBS-NIB 0 2709 Spain – IEO 0 15547 Sweden – SMHI 0 2600 Tunesia – INSTM 0 1497 Turkey – METU – IMS 0 2477 Ukraine – MHI 0 7476 United Kingdom – BODC 40122 40122 TOTAL 195131 341499

• Partners are now working with the new Mikado to upgrade V0 to V1

• The IIrd training workshop has been very useful for transferring expertise and instructing partners in use of CDI format and tools.

OBSERVATIONS



Summary

2006/2007

EDMED: 3.000

CSR: 38.525

EDMERP: 1.507

EDMO: 1.122

CDI: 195.131

2007/2008

EDMED: 3.500

CSR: 39.648

EDMERP: 1.600

EDMO: 1.134

CDI: 341.499

A very important increase in the CDI contents but extra coverage of national data is need

Documents

OBSERVATIONS & PRÉVISIONS CÔTIÈRES Metadata and Data Management Activities winthin NA 3-7 S. Iona (HCMR), D. Schaap (MARIS), L. Rickards