MAMA MD&IM Workshop - 28 January 2004

State-of-the-art tools and practices for marine data and information management –

a forward look

Lesley RickardsBritish Oceanographic Data Centre (BODC)

Chair, International Oceanographic Data and Information Exchange (IODE) Committee

MAMA MD&IM Workshop - 28 January 2004

What are we aiming for?• Simple access to all types of marine data (and

information) on an appropriate time scale• A virtual or distributed data centre• Data available on CD-ROM/DVD • End to end data management

Where are we now?• A collection of separate centres with a wide

variety of remits, skills and data• Made up of NODCs, RNODCs and WDCs• Sometimes work together in groups for

individual projects

MAMA MD&IM Workshop - 28 January 2004

MAMA MD&IM Workshop - 28 January 2004

Data Management and Communication System for the coastal module of GOOS

MAMA MD&IM Workshop - 28 January 2004

Data management practices:

• ‘Best’ rather than ‘state of the art’

• Compliance with IOC Data Exchange Policy

• Proper collaborative efforts, building on existing standards and practices – not reinventing the wheel each time a new project comes along

• Cooperation – partnership – collaboration

MAMA MD&IM Workshop - 28 January 2004

Timely, efficient and open access to the bestpossible data, metadata & associated products• Metadata Standards

• Discovery• Accompanying data (position, date/time, etc)• Data Documentation (qc history)

• Quality Control• Automatic tests• ‘Scientific’, delayed-mode

• Data dissemination• CD-ROM/DVD• On-line access

• Long term stewardship of data

MAMA MD&IM Workshop - 28 January 2004

Data management tools - METADATA• Metadata is all the descriptive information necessary to allow

users to find (discover), access, manipulate, process (request) and extract (recover) data, information and products

• Standards• ISO19115• Dublin Core • MEDI, GCMD, EDMED, FGDC, ANZLIC, (EDIOS), (CSR), etc, etc.

• Mappings between standards available (cross-walks)• ** Compliance with ISO19115 **• Various controlled vocabularies• Links with XML (schemas/DTDs)• MEDI Authoring Tool • All have search tools

• geospatial location, temporal information, keywords, controlled vocabulary items, “free text”

MAMA MD&IM Workshop - 28 January 2004

eXtensible Markup Language (XML)• XML is being widely used as a basis for both

dynamic web page development and more generally as a data exchange mechanism

• Data exchange aspects of XML include the ability to define flexible data structures that utilise the terminology of the subject area

• Data to be exchanged is packaged in a form more intuitive to the user.

• Extensive availability of free software for manipulation and transformation of the XML data stream

• Allows developers to easily develop, populate, exchange and transform data streams.

MAMA MD&IM Workshop - 28 January 2004

MarineXMLEU MarineXML

“… demonstrate how XML technology can be used to develop a framework that improves the interoperability of data for the marine community and specifically in support of marine observing systems. The project will develop a prototype of an XML-based Marine Mark-up Language (MML).”

ICES-IOC SGXML“… utilize or establish international standards to promote the seamless exchange of data from distributed data sources, by using a single parameter dictionary, well-defined and explicitly tagged metadata, and a common XML data structure, packaging all content and providing to the client datasets and software tools that are platform independent or web enabled”

MAMA MD&IM Workshop - 28 January 2004

Quality Control• (Real-time) automatic tests (but do not rely on them)

• Quality flags

Data visualisation tools• to include comparison with other data collected in the same place or

nearby, climatology; different ways of looking at the same data• Ocean Data View• Ncbrowse (for netCDF files)• EPIC (management, display and analysis of oceanographic and

meteorological data)

• Sea Level data • POL TASK2000 package + on-line tidal analysis• University of Hawaii JASL software• ESEAS – working towards new package

• Document QC (e.g. audit trail, data history)• Use existing standards/guidelines where available

• ICES data type guidelines, WOCE standards

MAMA MD&IM Workshop - 28 January 2004

Ocean Data View (ODV)• Interactive exploration and graphical display

of oceanographic and other geo-referenced profile, sequence or gridded data

• Runs on Windows (9x/NT/2000/XP), Linux, UNIX, and Mac OS X

• Data collection and configuration files are platform independent

• Interactive browse through large sets of station data

MAMA MD&IM Workshop - 28 January 2004

Ocean Data View (ODV)• High-quality station-maps, general property-

property plots of one or more stations, scatter plots of selected stations, property sections along arbitrary cruise tracks and property distributions on general iso-surfaces

• Display of original scalar and vector data by coloured dots, numerical data values or arrows

• Fast gridding algorithms allow colour shading and contouring of gridded fields along sections and on iso-surfaces

• Derived quantities calculated dynamically, displayed and analysed

MAMA MD&IM Workshop - 28 January 2004

MAMA MD&IM Workshop - 28 January 2004

Development of distributed systemsOPeNDAP (DODS)• Data servers for making local data accessible at

remote locations• Free software for download

Live Active Server (LAS)• Best for large, gridded environmental data sets• Dynamically generated graphics• Compare variables from distributed locations (using

DODS)

Thematic Real-time Environmental Data DistributedServices (THREDDS)• Access to large collection of real-time and archived

data sets from a variety of data sources• Analysis and display software

MAMA MD&IM Workshop - 28 January 2004

Other examples of distributed systems• US GLOBEC (US JGOFS)• NOAA Coastal Directory• Russian ESIMO• NERC Data Grid• IFREMER/SISMER Data Portal• SeaSearch Common Data Index• Distributed Generic Information Retrieval

(DiGIR) – protocol for retrieving structured information from multiple heterogeneous databases

• etc.

MAMA MD&IM Workshop - 28 January 2004

SISMERWEB portal

SISMERServer

Database« Data sets

catalogues »

Thematic database i

Thematicdatabase 1

SISMERServer Database

« Datasets catalogues »

SISMER WEB interface WEB interface i

Thematic database i

WEB interface 2

WEB interface 1

Thematicdatabase 1

NOW:

FUTURE: SISMER Web Portal

Automatic querying of existing systems

XML / ISO 19115 integrator 1 XML / ISO 19115

integrator i

XML / ISO 19115 integrator 2

Thematicdatabase 2

Thematicdatabase 2

SISMER Web Portal

MAMA MD&IM Workshop - 28 January 2004

ARGO STATUS(Jan 27, 2004)

1037 Active FloatsTarget: 3000 floats by 2006  

MAMA MD&IM Workshop - 28 January 2004

Argo Data Management• Data transmitted in real-time by satellite• Transferred to data centres• Messages decoded• Automatic real-time quality control tests• Data passed to global centres (GDACs) for

dissemination• Delayed-mode quality control and calibration• Replacement version sent to GDACs• Regional centres for further quality control

and products

MAMA MD&IM Workshop - 28 January 2004

Location of Argo float profiles

01 – 27 Jan2004

No. of profiles:2139

Data available for download:

ProfilesTrajectoryMetadataTechnical info

From:• US GODAE• CORIOLIS

MAMA MD&IM Workshop - 28 January 2004

• Web-based provider• Global geo-referenced information• Accurately identified marine species• Expert species level and habitat level

databases• Variety of spatial query tools for visualizing

relationships among species and their environment.

• Strives to assess and integrate biological, physical, and chemical oceanographic data from multiple sources.

• Users, including researchers, students, and environmental managers, gain a dynamic view of the multi-dimensional oceanic world

Part of the Census of Marine Life (CoML)

Ocean Biogeographic Information System (OBIS)

MAMA MD&IM Workshop - 28 January 2004

Gulf of Maine Biogeographic Information System(GMBIS)

• Regional implementation for CoML

• Partner with OBIS

• Integrated into the Gulf of Maine Ocean Observing System (GoMOOS)

• Designed to assimilate and integrate marine ecosystem and fisheries data, as well as natural-history information

• Included is an advanced oceanographic geographic information system (GIS)

MAMA MD&IM Workshop - 28 January 2004

Gulf of Maine Biogeographic Information System(GMBIS)• Access to biological, physical, chemical and geological

data and information• Enhance understanding of biological patterns and their

changes through time• “Aggregation server” providing access, rapid

visualization and data download capabilities• Server will rely on a combination of archived (local

access) data as well as dynamic access to remote data providers

• Visualisation and other interactive software will be designed to help the user evaluate what data are available, combine data layers and download data

MAMA MD&IM Workshop - 28 January 2004

Gulf of Maine Biogeographic Information System (GMBIS)

Access to:• historical data• taxonomic collections• geological base maps• modern remote sensing data

(buoys, CODAR, satellite)• modelling products• broad-scale survey data (e.g.

living marine resources)• other monitoring programs

(e.g. COASTWATCH, Continuous Plankton Recorder)

MAMA MD&IM Workshop - 28 January 2004

GMBIS uses Environmental Analysis System (EASy)

• Storage, dissemination, analysis, integration, and dynamic display of spatially referenced series of oceanographic data

• PC-based• Aids interfacing of multivariate oceanographic data• Both data and model outputs can be imaged in time through

diverse kinds of displays• Including vector, contour, and false-colour imagery

• Vertical structures can be depicted along line transects or point-sampling stations

• Time series can also be visualised• Patterns in the spatial distribution or organisms to be visualized

and compared to other spatial distributions• Including both biological and other oceanographic variables• Even when characterised by different scales of sampling and

different degrees of resolution

MAMA MD&IM Workshop - 28 January 2004

CONCLUSIONS• Use of modern IT techniques in a transparent

manner to improve service to users (internet, web, distributed systems, XML, etc.)

• Setting the standards (metadata, data quality control, data stewardship)

• Working collaboratively whilst responding to national remit

• Increasing data diversity (many different parameters being measured)

• Developing systems to deliver (near) real-time data