PLATA-HISHydrologic Information System Central Web Service Registry

for the Plata basin in South America

Technical proposal

Silvano PecoraHead of Hydrology – ARPA Emilia-Romagna, Italy

November 2015

2

Table of Contents

1. Data collection and dissemination in hydrology ............................................................................. 32. Sharing of data and information .................................................................................................... 43. A federated approach to water data sharing.................................................................................. 5

3.1. The HIS architecture .................................................................................................................. 63.2. The HIS components ................................................................................................................. 73.3. Standardized service types and data formats .......................................................................... 11

4. WIS coompliance ........................................................................................................................ 125. Resource issues ......................................................................................................................... 12

3

1. Data collection and dissemination in hydrology

The ultimate goal of data collection in hydrology, be it precipitation measurements, water-levelrecordings, discharge gaugings, groundwater monitoring and water quality sampling, is to provide aset of sufficient good quality data that can be used in decision-making in all aspects of waterresources management, in the wide range of operational applications as well as in research.Decisions may be made directly from raw data measurements or based on derived statistics or on theresults of many stages of modelling beyond the raw data stage, but it is the collected data that formthe basis for these decisions. Data sets are of great intrinsic value as they are collected through ahuge commitment of human and financial resources and often during a long period of time, and theyacquire also a use value when are made available in an usable form to the final users to respond totheir specific needs.The management of hydrological data is therefore important work in itself and this work must beperformed effectively in order to maximize the results of the investments put in their collection andtheir possibility of being effectively used. WMO Hydrology and Water Resources Programme hasalways given high priority to issue related to data management, be it through the work of experts of theCommission for Hydrology, the publication of manual and other guidance material, the implementationof projects for improving data collection and information production (such as WHYCOS) and datarescue, the promotion of the access to and exchange of hydrological data, through the implementationof the Resolution 25 (Cg-XIII) - Exchange of Hydrological Data and Products.In support of these activities of the access to and exchange of hydrological data provided by theNational Hydrological Services belonging to the Plata basin (namely, Argentina, Bolivia, Brasil,Paraguay and Uruguay), this technical proposal is intended to provide additional operationalcapability, in particular for in situ water observations, to be called PLATA-HIS, as a federated resourcefor National Hydrological Services.

Figure 1 – Discovery and access of hydrological data provided by the National Hydrological Services of the Plata basin

4

The scope of the current proposal is defined as a global registry of water data and map servicescatalogued using the standards and procedures of the Open Geospatial Consortium and the WorldMeteorological Organization. This registry as a specialist hydrological component in the WMOInformation System (WIS) will be open to all users and institutions from any country or level ofgovernment, and applies to any type of water information. It is anticipated that this activity will beundertaken in collaboration with Theme 2: “Data Operations and Management” of the WMOCommission for Hydrology, which is active from 2012 to 2016.

2. Sharing of data and information

Resolution 25, adopted in May 1999 by the Thirteenth World Meteorological Congress, commits WMOMembers to broaden and enhance, whenever possible, the free and unrestricted internationalexchange of hydrological data and products. This engagement is an important contribution to disasterrisk reduction, human safety and well-being, and shared socio-economic benefits. In 2005, a GlobalClimate Observing System (GCOS) report identified standards as a key problem in data exchange inglobal hydrological and atmospheric networks. Thus, in 2009, WMO and the Open GeospatialConsortium (OGC) jointly formed a Hydrology Domain Working Group (HDWG) to address this issue.The OGC adopted WaterML 2.0: Part 1 - Time Series in 2012 and Part 2 – Rating Curves, Gaugingsand Cross Sections in 2015 as official standards (see www.opengis.net/doc/IS/waterml/2.0). Encodedin eXtensible Markup Language (XML) and based on existing OGC standards, WaterML 2.0 providesan interoperable hydrological exchange format that may be used to address a wide range of userneeds. These include the exchange of data relating to:- In situ observations at hydrological (gauges, reservoirs) or climatological stations;- Forecast products (probabilistic or deterministic time series) at forecast locations;- Emergency or operator-oriented alerts (of threshold exceedance) and reports;- Time series of planned intake and release/discharge;- Groundwater observations of the water level within wells.Using WaterML 2.0, it is possible to link together local, national, regional and global water informationsources as part of global water information networks. The availability of such a format will alsofacilitate the usability of data shared through the WMO Information System (WIS).The 2012 WMO Commission for Hydrology (CHy) recommended that WMO Members test, throughpilot projects during 2013-2016, the use of WaterML 2.0 for the exchange of hydrological data with aview of its adoption as a joint WMO/ISO standard. The Commission also noted the importance ofWaterML 2.0 and other emerging OGC standards for improving service delivery in key areas, includingthe WMO World Hydrological Cycle Observing System (WHYCOS) and the WMO Flood ForecastingInitiative.In 2014 the CHy Advisory Working Group has proposed the WMO Hydrological Observing System(WHOS) as the means to provide the most comprehensive hydrological component in fulfillment of theWIGOS objective of “an integrated, comprehensive, and coordinated system which is comprised of thepresent WMO global observing systems”. WHOS is conceived as a portal to facilitate access toalready available on-line real-time and historical data, drawing from the water information systems ofcountries around the world that make their data freely and openly available, including HYCOSprojects. WHOS is composed by two phases: Phase 1 provides a map interface with links to thoseNHSs that make their real-time and historical stage and discharge data available online (initialimplementation July 2015); Phase 2 provides a fully WIS/WIGOS compliant services-orientedframework linking hydrologic data providers and users through a hydrologic information systemenabling data registration, data discovery, and data access (Beta version for CHy-15 review andendorsement in December 2016 and initial implementation for EC approval in June 2018).

5

Figure 2 – WMO Integrated Global Observing System (WIGOS)

3. A federated approach to water data sharing

PLATA-HIS aims at developing CyberInfrastructure and services to support the advance of hydrologicscience in the international hydrological community. Hydrologic information science involves thedescription of hydrologic environments in a consistent way, using data models for informationintegration and software tools that are either customized or based on commercial products to directlyinteract with the deployed data storage and communication components.PLATA-HIS will support storage of water observations data in a relational database, publication viaweb-services on the internet, federation with water observations data published by water agencies,and search across the various data holdings in the network. These advanced data access andanalysis capability are provided through the use of web services using a standardized data formatsand service types.

6

a) Hydrologic data request b) Hydrologic data collection

c) Hydrologic data transformation d) Hydrologic data delivery

Figure 3 – Standardized data formats and service types in hydrologic data exchange

3.1. The HIS architectureWhen sharing information across scientic communities, it becomes important to dene a standardframework through which large quantities of multidisciplinary information can be shared, discoveredand accessed.Utilizing a collection of WMO and OGC Web services, PLATA-HIS is designed as a “services stackframework” that shares catalog data, metadata and data with the user. The services stack frameworkidenties three types of services as essential to sharing water information across the Web: catalogservices, metadata services and data services. These three services work together to completelyindex, describe and provide access to water information (e.g. time series). Catalog services provideusers with an index of hydrologic metadata, metadata services identify collections of time seriesavailable over a domain of space and time, and data services provide the user with the raw data for aspecied temporal period and spatial area. This framework is designed to publish and distribute timeseries, but it can be extended to include grids and coverages as well.While other interoperability studies have focused on implementing custom data streams (e.g. bridges,adaptors, etc.) between clients and server interfaces, PLATA-HIS focuses on a common data andmetadata management model that leverages a suite of WMO and OGC standard Web services whichcan be applied to multiple scientic communities – in particular hydrologic and atmospheric sciences.

7

Figure 4 – Discovery and access functionalities supported by broker and mediator

Furthermore, PLATA-HIS can be integrated within existing data discovery frameworks (e.g. portals,gateways, etc.) by leveraging mediation and brokering services.

3.2. The HIS componentsAs with any other Service Oriented Architecture, PLATA-HIS is built around two fundamentalcomponents: (1) service providers and (2) service consumers. Although service consumers directlyconnect to service providers to request and receive data, a third component, a service registry, isintroduced to facilitate the discovery of different service providers; this can be done using variouskeywords, metadata and lters. As service providers introduce their services within PLATA-HIS,services are registered at the service registry. Service consumers can then search the registry to ndavailable services of interest.PLATA-HIS can be dened as a collection of components which work together to store, index, accessand distribute hydrologic information. The system contains servers, catalogs and applications whichcommunicate with one another through a set of web services. Web services are the set of protocolsand specied functions that exchange hydrologic metadata and data (i.e. time series) through the webusing a common standardized language,The individual components of PLATA-HIS each serve an important role in the data discovery andfetching process. Data providers are the principal locations for storing large volumes of hydrologicdata, specically time series. Within the provider itself, data and metadata are managed in a databaseand then exposed through its Principal GISC (Brasilia) as well as through a suite of web services sothat remote users can then access the data through the web.

8

Figure 5 - Data provider in the Plata basin

Another component of PLATA-HIS is the hydrologic metadata catalog. The registry is the componentof PLATA-HIS which facilitates the discovery of hydrologic data that has already been published bythe data providers as required in the Manual on WIS (WMO No.1060) section 3.6.8 “Describeinformation with metadata”.Within PLATA-HIS, data providers are the primary repositories for hydrologic data, while the registry isthe primary repository for hydrologic data services. The registry provides an interface where users cansearch registered data services by specifying keywords and metadata which describe the hydrologicdata of interest. The registry is like a Google for discovering hydrologic time series information. Datapublishers can register their data on the registry and provide brief descriptions of the datasets theywant to share. This is an important aspect of PLATA-HIS because it allows for data to be organizedand discovered in an efficient, structured and methodical process.

9

Figure 6 - Registry applied with a federated approach to the Plata basin

The third and final component of PLATA-HIS is the web or desktop client. The client is the componentof PLATA-HIS that allows for the harvesting of hydrologic information at the locality of one’s owncomputer or analytical system as per Manual on WIS (WMO No.1060) section 3.6.7 “Provide accessto information”. The client is a platform located on the user’s machine and communicates with bothdata services and the registry. Users can directly download hydrologic information from data providersif they already know of their existence or can search the registry for data that they might not knowabout. Once the data of interest has been discovered, users can download the information onto theirlocal databases.

10

Figure 7 - Desktop client searching and downloading data via web services from the Plata HIS

Figure 8 - Web client searching and downloading data via web services from the Plata HIS

11

3.3. Standardized service types and data formats

There are three components to the services stack framework which work together to provide a systemin which data consumers can readily discover and access both time series and coverage data usingspatial, temporal and semantic lters: Catalog Services, Metadata Services and Data Services.At the core of the services stack framework lie the metadata services which act as middlewarebetween the catalog services and data services. Data services ultimately provide the user with thedata they are searching for, whereas catalog services allow users to perform federated searchesacross multiple data providers. Metadata services link both these layers together by being registeredat the catalog level and providing all the information needed to access information at the data level.Data services are at the bottom of the services stack framework and are responsible for providingusers with the data sought after. These services follow the Sensor Observation Service (SOS) andWeb Coverage Service (WCS) specications outlined by the OGC. For hydrologic information, timeseries data is obtained through a SOS implementation encoded in WaterML 2.0.Similarly, for atmospheric science information, coverages can be obtained through a WCSimplementation as a netCDF le. Each service allows users to specify spatial, temporal and semanticlters to obtain subsets of the larger datasets supplied by the data service.Metadata is an essential component of the data sharing process. Not only does it facilitate the searchand discovery of information within one’s own research community but also helps fosterinteroperability between research communities. Metadata is used to describe a set of data that share acommon ground to others who are not directly familiar with the information.The metadata issue is very well documented and has led many scientic communities to adoptmetadata standards created by the International Organization for Standardization (ISO). Specically,the ISO-19115 standard provides a manner in which geographic metadata can be published acrossweb based information systems.

Figure 9 – Catalog interfaces in the Plata HIS

12

Catalog services aid in the management, discovery and distribution of metadata describing geographicdatasets and services. Within the PLATA-HIS services stack framework, catalog services function asthe interface through which data consumers discover indexed metadata services published as WFS.As part of the standard suite of OGC services, Catalog Services for the Web (CSW) are the Webservices that focus on the management and indexing of geographic metadata. The OGC designedCSW to help data consumers search through a set of matching resources. As such, CSW allow datapublishers to register and index a set of metadata services with a variety of different metadata prolesas dened by the ISO (e.g. ISO 19115/19119). Using a CSW interface, data publishers are able toshare their catalog of metadata with search clients as well as other catalogs.

4. WIS coompliance

The WMO Information system (WIS) is the single coordinated global infrastructure responsible for thetelecommunications and data management functions. It is the pillar of the WMO strategy for managingand moving weather, climate and water information in the 21st century. WIS provides an integratedapproach suitable for all WMO Programmes to meet the requirements for routine collection andautomated dissemination of observed data and products, as well as data discovery, access andretrieval services for all weather, climate, water and related data produced by centres and Membercountries in the framework of any WMO Programme.WIS was designed to dramatically extend WMO Members' ability to collect and disseminate data andproducts. Owned and operated by Members, it will be the core information system utilized by theWMO community, providing linkages for all WMO and supported programmes associated withweather, climate, water, and related natural disasters.In the context of its role in WIS, PLATA-HIS will publish discovery metadata in the GISC Brasiliaallowing WMO users to access a range of services on PLATA-HIS. By publishing in GISC Brasilia, thediscovery metadata will also be available to the GEOSS community through the GEO Data Portalwhich includes all WMO discovery metadata.A potential future phase of the PLATA-HIS is to pilot possible extensions of WIS functions andservices tailored to the needs of the hydrological community, but potentially beneficial to otherprogrammes, including databases, simulation models, web-GIS and other tools.

5. Resource issues

This technical proposal will be undertaken within existing resources. However, it is recognized thateventually the water data sharing system could require additional resources to ensure sustainedtechnological support, and to expand the objectives of the proposal which could include the collectionand sharing of additional parameters, as well as the development of models to support delivery ofproducts and services through related WMO efforts.