The Color of Ocean Data, Brussels, Belgium, 25-27 Nov, 2002The Color of Ocean Data, Brussels, Belgium, 25-27 Nov, 2002

Experience of XML use in Russian NODC and perspectives - MedBlackDODS and other

Nicolay Mikhailov, Evgeny Vjazilov, Alexander Besprozvanykh, Natalja Puzova

RIHMI-WDC/Russian NODCObninsk, Russian Federation

CONTENTCONTENT

1, Two words about XML1, Two words about XML

2, Russian NODC experience2, Russian NODC experience

3, Marine XML 3, Marine XML

4, View on Distributed Ocean Data System4, View on Distributed Ocean Data System

What is XML (What is XML (eXtensible Markup Language) ?- eXtensible Markup Language) ?- classicclassic

XML XML is a component of Web-technologies to describe is a component of Web-technologies to describe information resources for exchange and distribution information resources for exchange and distribution

under Webunder Web

Independence from computer platformsIndependence from computer platforms

Support by the developersSupport by the developers – – W3C standard for HTTPW3C standard for HTTP

Self-describingSelf-describing - - readablereadable and allows and allows to automate data processingto automate data processing

Hierarchy-Hierarchy-allows easily to describe allows easily to describe complex structures of the datacomplex structures of the data

This is sintactic tool onlyThis is sintactic tool only

It is required to It is required to rewrite rewrite existing existing software software - - partiallypartially

Data volume and processing Data volume and processing timetime

More complicated technologyMore complicated technology

New level of marine DM demands the New level of marine DM demands the use use of new of new information information technologiestechnologies

Why XML?Why XML?Computer network

Web midleware

Web DBMS/GIS

Web DM applications

XML roleXML role

Driving forcesDriving forces – national research programme World Ocean – national research programme World Ocean

10 ministries/agencies, 40 partners

1999-2002 – 1-st stage, 2003-2007 – 2-nd stage, 2008-2012 – 3-d stage

Unified System of Information on World Ocean ConditionsUnified System of Information on World Ocean Conditions

Marine Marine environmentenvironmentMeteo, Physic Ocean,Meteo, Physic Ocean,

Ice, Chemistry, Ice, Chemistry, Pollution, Pollution,

Bioresources, Bioresources, Geology-geophysic Geology-geophysic ….….Marine activityMarine activity IInfrastructurenfrastructure

Lows and Lows and specificationsspecifications

Social economic ,...Social economic ,...

Users:Users:

Federal levelFederal level

Regional level Regional level

Organization levelOrganization level

ESIMO ESIMO ProductsProducts

•MetadataMetadata

•Observ. dataObserv. data

•Climat, Climat, analisys, analisys, forecastsforecasts

•Analytical Analytical informationinformation

•forfor •requestsrequests•““pushing”pushing” •special services special services

•In In – – operational operational and delay-mode and delay-mode

Collection

Accumulation Processing

Dissemination

Integration and services

Storage

1. Historical data - Russian NODC (1920-1984)

2. Historical data - ICES (1985-1986)

3. GTSPP data (Aug -Sept 2000)

Web-serverApp server

5. GTS Data • SHIP• BATHY• TESAC• BOUYfor Aug-Sept. 2000

DBMSRIHMI-WDC Obninsk

Web/XML technology prototype – IODE XVI Demonstrator/2000

Web-serverApp server

Lisbon, DM 1

Lisbon, DM 2

Lisbon, DM 3Application

• Caspian Sea, 20022002/2003/2003

ГосНИНГИГосНИНГИ

ЦКБ ГМПЦКБ ГМП

НИЦ НИЦ ПланетаПланета

ВНИИГМИ-МЦДВНИИГМИ-МЦД

ГРМЦГРМЦ

• Black Sea,, 2000 г. 2000 г.

RTS

Integrated DBHist. DB

Sat. DBSat. DBClimat DB

GTSGTS

GTS DB

ESIMO integration module on Web/XML technology prototype – 2000/2002

Web/XML “live” applications/2002Web/XML “live” applications/2002

ESIMO Integration Module – ESIMO Integration Module – metadata and information metadata and information management ( as information resource objects)management ( as information resource objects)

Cruise and ocean stations metCruise and ocean stations metааdatadata

On-line GTS data base – ocean and marine On-line GTS data base – ocean and marine hydro/meteo ship/coastal data hydro/meteo ship/coastal data

•MEDAR/MEDATLAS II InventoryMEDAR/MEDATLAS II Inventory•Russian NODC/WDC-B on-line inventoryRussian NODC/WDC-B on-line inventory

ESIMO Integration Module – Web-portal /2002ESIMO Integration Module – Web-portal /2002

ESIMO centres

Web - портал Главная страница - список ИП

Global Inventory of information Global Inventory of information resources resources

HtmlHtml GIFGIF ApplicationsApplicationsDocDoc DB/DB/filesfiles

ftpftp

• Information resourcesInformation resources

•Web-sites with Web-sites with metadata, data and metadata, data and productionproduction

•Independed Independed applicationsapplications

• Resource descriptionsResource descriptions

•Identification and Identification and typetype

•ParametersParameters

•Time and geoTime and geo

• AuthorsAuthors

•Links to XML docs and Links to XML docs and etc. etc.

ServicesServices

•User access User access passwords passwords

•Check of reCheck of resourcesource

readiness readiness

•User User WP WP Adjustments Adjustments

•......

Testing: 20.09.2002 – 30.11.2002Testing: 20.09.2002 – 30.11.2002

Launching: 20-25.01.2003Launching: 20-25.01.2003

Specialized Specialized ApplicationsApplications

Object name Object amount

XML-file name

Data Provider Descriptions Marine organizations 100 orgs.xml

Projects and Programmes 90 project.xml Experts 110 experts.xml

Observation Networks 15 observ.xml Observation Methods 25 Metods.xml

Research Vessels 800 ships.xml Satellites 35 satel.xml

Observ. Equipments 150 prib_CCB.xml Coastal Stations 700 Coastel.xml

Information Resource Descriptions Data sets and databases 400 Data_set.xml

R/V ocean cruises 33500 Cruise.XML Marine maps 160 maps.xml

Web-resources 20 web_res.xml DM tool descriptions

Data Formats 25 format_n.xml Software 20 soft.xml

Parameter Dictionary 400 UDOP.xml

Metadata on ESIMO Web-pagesMetadata on ESIMO Web-pages

< ST_LIST REF = "TN89196300001"OBS = "bottle" xmlns:dt="urn:uuid:C2F41010-65B3-11d1-A29F-00AA00C14882/">< ST NUM = "00091" ><LAT> 41.5 </LAT><LON> 30.8333 </LON>< DAT dt:dt = "date" > 1963-04-11 10:00 </DAT><GOR> 1233 </GOR><LIST_PAR><PAR> PRES </PAR><PAR> TEMP </PAR><PAR> PSAL </PAR><PAR> DOX1 </PAR></LIST_PAR></ST>

Cruise and ocean metadataCruise and ocean metadata - - MEDAR/MEDATLAS Inventory and on-line MEDAR/MEDATLAS Inventory and on-line Russian NODC/WDC-B inventoryRussian NODC/WDC-B inventory

<CR_LIST xmlns:dt="urn:uuid:C2F41010-65B3-11d1-A29F-00AA00C14882/"><CR><CNT> 06 </CNT><SHIP> 06MT </SHIP><REF> AD06199325003 </REF><DC> FI </DC><FILES>< F OBS = "ctd" INV = "st.mete25c.ctd.xml" name ="/Ctd/new/mete25c.ctd " ><IHB> 28B </IHB>< ST dt:dt = "date" > 1993-07-03 </ST>< FN dt:dt = "date" > 1993-07-16 </FN></F></FILES>< R / ></CR><CR>

DBMS+Data files

XML filesXML files Java Java

MEDAR/MEDATLAS IIMEDAR/MEDATLAS IICDsCDs

Web-serverWeb-server

on-line Russian on-line Russian NODC/WDC-B NODC/WDC-B

inventoryinventory32,8 thou. cruisec32,8 thou. cruisec

COD Conference, Brussels, 25-27 Nov, 2002

MEDAR/MEDATLAS Cruise InventoryMEDAR/MEDATLAS Cruise Inventory

On-line GTS Data Base - HYDROMETEO DTDOn-line GTS Data Base - HYDROMETEO DTD

<DataBlock dbI="dbOcean"> <Table TbN="Buoy_Lv"> <Data StI="21533" Lat="36.2" Lon="130.47"> <D T="1999-12-06 22:22:00.0" Lev="0.0"> <P N="TYPE_OBS" V="74" /> <P N="MARS10" V="131" /> <P N="CENTR" V="WKARS" /> <P N="LEVTYPE" V="100" /> <P N="TEMPWAT" V="16.36" Q="0" /> </D> <D T="1999-12-06 22:22:00.0" Lev="3.0"> <P N="TYPE_OBS" V="74" /> <P N="MARS10" V="131" /> <P N="CENTR" V="WKARS" /> <P N="LEVTYPE" V="100" /> <P N="TEMPWAT" V="16.68" Q="0" /> </D> </Data>

</Table></DataBlock>

Four forms are used for parameter definition: Form 1 - pointer to DBMS table name and column name

<ElT TbN="Buoy_Lv" ClN="TEMPWAT"/>Form 2 – code parameters:

<ElC Cod="2102"/>, Cod – from Parameter DictionaryForm 3 - function to produce parameter:

<ElF FnN="Function_Name"/>Form 4 - defining the process of calculation.

Data are placed in DataBlock Key values: Station ID, Latitude and LongitudeD node – data segment (profile)P node – data string, where N - temporary name, V - value, Q -quality control flag, D - additional definition.

DBMS

App Server MetDD DOM App Server MetDD DOM

Web-serverWeb-server

GTS

Generalization of XML useGeneralization of XML use

Common data model – structured presentation of application domain objects and relations

SemanticSemantic SyntaxSyntax

Common Parameter Dictionary – Terms and parameters definitionefinitions, s, Other attributes Other attributes

Common Codes – Platforms, Organizations, Geoareas, Equipments, QC sighs, Units, ...

Request model – structured presentation of data and services requests

Tags, attributes -start and end of element, hierarhy definition, ...

Entity references - start and end of element, hierarhy definition, ..

Processing instruction - specific information for applications, ..

Character data sections (CDATA) - character information

Document type declaration - DTD construction

DTD XML Scheme

New is well well forgotten oldNew is well well forgotten old - GF-3 - XML syntax, GF-3 subsets - - GF-3 - XML syntax, GF-3 subsets - DTD, GF-3+GF-3 Proc - DOM DTD, GF-3+GF-3 Proc - DOM

Format Name

Obs. type

Meta data

Codes

Data Struc-turing

Soft-ware

Self-defin-ing

ICES H, CTD CI WMO, IOC,

ICES FPAR Ocean

PC No

ICES Blueprint '86

H, CTD CG GF3 VPAR OceanPC

Yes/GF3

GF3 (Subsets) M, H, MBT/XBT, CTD,…

CR, CG, OM WMO, GF3, IOC VPAR GF-3 Proc

Yes/GF3

JGOFS M, H, MBT/ XBT, CTD, Bl, PL

CR, CG, OM WMO, GF3, added

VPAR W/R Yes/GF3

GTSPP M, H, MBT, XBT, CTD CI, OM, SQ, HQ, HM

WMO, IOC, GF3, MEDS, NODC

FPAR W/R No

BUFR (subsets) Bathy, Tesac CG WMO VPAR W/R Yes/BUFR

NODC (SD2) M, H CR, CG, OM WMO, IOC, NODC

FPAR ? No

NODC (BT) MBT, XBT

CR, CG, OM NODC FPAR ? No

NODC (h/r STD/CTD)

CTD CR, CG, OM WMO, IOC, NODC

FPAR ? No

OCL (WAOD’98) H, MBT/XBT, CTD, BL CI, OM, HM WMO, NODC/OCL

VPAR R Yes/?

TOGA M, H, XBT, CTD, … CR, CG, OM WMO, IOC, NODC, added

FPAR ? No

WOCE M, H, XBT CTD,… CR, CG, OM WMO, IOC, NODC, added

FPAR ? No

GLOSS Sea level CR, CG, OM added FPAR R No MEDS M, H, MBT/XBT CTD CG, OM, SQ,

HQ IOC, WMO, GF3, MEDS

VPAR W/R No

MEDATLAS H, MBT/XBT, CTD CR, CG, OM, HQ

IOC, WMO, GF3, ICES

VPAR QC Medar

Yes/GF3

P-3 (NODC OPDB)

M, H, MBT/XBT CTD CR, CG, OM, HQ

WMO, NODC VPAR DBMS Yes/?

NODC Russia M,H, MBT/XBT, CTD, PL

CR, CG, OM WMO, IOC, National

FPAR

HDDL System

Yes/HDDL

NODC Argentina /RNODC

M, H, MBT/XBT, CTD CG, OM WMO, IOC, National

FPAR

W/R No

NODC Spain M, H, MBT/XBT, CTD CG, OM WMO, IOC, National

FPAR

W/R No

OceanAtlas H, CTD CR No FPAR OA No TSV-O(DVO) H, CTD CR No FPAR DVO No

Ocean Data Management

XML1

XML2

XML9

XML8

XML13

XML

XML10

XML21

XML14

XML22

XML15

XML100

XMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXMLXML

Stages of Marine XML development – Russian NODC viewStages of Marine XML development – Russian NODC view

Blocks or Bricks (Bob Keely, MEDS) development

Common Data model

Relations (int. and ext.) development

Combining and normalization

Common Parameter Dictionary adoption

Data model description using common Dictionary

Request model

Common code tables adoption

Mapping on XML syntax - Marine XML Scheme

Main Aspects of Distributed Ocean Data Base issues

Technical issues

Funcitional issues

DODS technical issues

Applications

Integrators

Navigator (s)

Data Sources

Marine XML:-common data and request model - Common dictionary and Code-full XML Scheme

Middleware:- Web-server,- App Server- local DM systems

Applied software- single user entry- user interface for criteria and results- “puch” and “requiest/replay” modes

System software:- Navigator (user requiest parsing, requiest navigation, data source monitoring, duplication, …) - Integrator (DBMS, File system, Web-sources converting in Marine XML

Communications:- Internet based- routes and providers

W3C standards

Web server

Web-portalNavigation/Processing

DODS functional issues – Mediterranean/Black seas case

Discipline Discipline XML XML filefile

DBDB11, DB2, DB3,…, DB2, DB3,…

DS1DS1-SISMER-SISMER

PartnersPartners

DS5 -DS5 - RNODC RNODC

DS2DS2-IEO-IEO

DS4 -DS4 - HNODC HNODC

Web serverWeb server

Integrator

DB

Web serverWeb server

Integrator

DB

Web serverWeb server

Integrator

DB

Web serverWeb server

Integrator

DB

DS3 -DS3 - OGS OGS

Poly-discipline Poly-discipline XML fileXML filess

DBDB11, DB2, DB3,…, DB2, DB3,…

Operational data Operational data XML fileXML file

DBDB11, DB2,…, DB2,…

Web serverWeb server

Integrator

DB

Collection Accumulation Processing

MFSSMFSSArgo/Caeiolis/Argo/Caeiolis/PaseidonPaseidon GTSGTS Obs.dataObs.data ClimatClimat

Storage

The Color of Ocean Data, Brussels, Belgium, 25-27 Nov, 2002The Color of Ocean Data, Brussels, Belgium, 25-27 Nov, 2002

Experience of XML use in Russian NODC and perspectives - MedBlackDODS and other

Nicolay Mikhailov, Evgeny Vjazilov, Alexander Besprozvanykh, Natalja Puzova

RIHMI-WDC/Russian NODCObninsk, Russian Federation

History of new technologies (XML and etc.) use History of new technologies (XML and etc.) use in Russian NODC/RIHMI-WDCin Russian NODC/RIHMI-WDC

1996-1998 - 1996-1998 - aaccumulation of knowledge/experience and development ccumulation of knowledge/experience and development of the principal of the principal decisions on client-server DM technologydecisions on client-server DM technology

1990-1995 - 1990-1995 - DBMS/GIS use on personal/organization level DBMS/GIS use on personal/organization level

1999-2001 – 1999-2001 – development of technology prototypes anddevelopment of technology prototypes and design design decisions on distributed marine data system as unified decisions on distributed marine data system as unified information space/data warehouseinformation space/data warehouse

2002-….. – 2002-….. – development of “live” DM applications under development of “live” DM applications under Web/XML technology andWeb/XML technology and design decisions on components of design decisions on components of distributed marine data systemdistributed marine data system

Driving forcesDriving forces – national research programme World Ocean – national research programme World Ocean

END