© Crown copyright Met Office CF-netCDF in WMO Bruce Wright (Senior IT Architect), 7 th GO-ESSP Community Workshop, 17-19 September 2008

© Crown copyright Met Office

CF-netCDF in WMOBruce Wright (Senior IT Architect), 7th GO-ESSP Community Workshop, 17-19 September 2008


Contents

This presentation covers the following areas

• WMO

• ET-ADRS

• ET-ADRS Initial Meeting

• Since the ET-ADRS Initial Meeting

• Some Thoughts

• Discussion


WMO(World Meteorological Organization)

http://www.wmo.int/pages/index_en.html


World Meteorological Organization (WMO)

• Specialized agency of the United Nations (UN)

• Membership of National Meteorological and Hydrological Services (NMHSs)

• Facilitates the free and unrestricted exchange of data and information, products and services in real- or near-real time on matters relating to safety and security of society, economic welfare and the protection of the environment

• Recognized as international standardization body by ISO


WMO Structure

• Commission for Basic Systems (CBS)

• Information Systems & Services (ISS)

• Data Representation Codes (DRC)

• Assessment of Data Representation Systems (ADRS)

Technical Commission (8)

Open Programme Area Group (OPAG) (4)

- Supported by Implementation Coordination Team (ICT)

Expert Teams (ET)

- Permanent: Maintains the WMO code forms (e.g. GRIB, BUFR)

- Temporary: Deliver recommendations


ET-ADRS(Expert Team on Assessment of Data Representation Systems)

http://www.wmo.int/pages/prog/www/WDM/Et-adrs.html


ET-ADRS Formation

• Extraordinary session of CBS (Seoul, Nov 2006):

• Agreed to study the implications of using data forms such as XML or netCDF for meteorological data, especially for operational real time exchanges

• Requested the establishment of the ET-ADRS within the OPAG-ISS for this purpose

• All technical commissions were invited to participate

• Charged the ET-ADRS to provide recommendations, including a proposal on CBS Policy

• ET-ADRS to submit outcomes to the ICT of the OPAG-ISS for Nov 2008 meeting (ahead of next CBS session in Mar 2009)


ET-ADRS Aim

• Assess advantages and disadvantages of different data representation systems, e.g.:

• GRIB / BUFR / CREX

• XML

• NetCDF / HDF

• For use in:• Real time operational international exchanges between

NMHSs

• Transmission of information to users outside the NMHSs

• Including implications:• Need for defining standardization

• Data processing development and integration

• Costs and benefits: flexibility, compression, feasibility of implementation, etc


ET-ADRS Membership

• Co-Chairs

• Simon Elliott (Eumetsat), Fred Branski (NOAA)

• Core members (6 – 4 at meeting)

• Also presented on data representation systems (i.e. GRIB/BUFR/CREX, XML, netCDF, HDF5 & ASN.1)

• Associate members (31 – 3 at meeting)

• Technical Commissions (4 reps at meeting)

• Invited Experts (8 at meeting)

• International Organizations (1 ICAO rep at meeting)


ET-ADRS Initial MeetingSilver Springs, Washington, 23-25 April 2008

http://www.wmo.int/pages/prog/www/WDM/ET-ADRS-1/Documents.html


ET-ADRS Initial Meeting

• Review of data representation systems:

• Presentations & synopsis documents (Core team members)

• Including SWOT analysis & recommendations

• Discussion

• Comparison of data representation systems

• Sub-groups

• Particular criteria

• Plan for future work


CBS Policy Proposals (Initial Meeting Report, version 7)

• Data Representation Systems (DRSs) should be driven by user requirements

• Consider the need for interoperability when different DRSs are being utilized

• Application of ISO191xx series of geographic information standards to the development of a WMO conceptual model of data representation:

• Standard approach -> WMO core profile, including feature catalogues, application schemata and data specifications

• Facilitating interoperability and data exchange between applications based on DRSs associated to BUFR, CREX, GRIB, XML, netCDF and HDF


Recommendations 1 (Initial Meeting Report, version 7)

• Adoption of netCDF where fit for purpose

• Adoption of NetCDF-4 (rather than NetCDF-3):

• Improved performance

• Convergence of NetCDF and HDF (effectively adopting 2 standards though one API).

• It was considered important do distinguish between recommending netCDF as a format and recommending the use of the API from Unidata


Recommendations 2 (Initial Meeting Report, version 7)

• Noting that:• NetCDF itself provides very limited metadata• CF metadata convention has been developed for

sharing climate model and NWP forecast data.• Although used in some operational meteorological

implementations, there are shortcomings with the CF

• Engagement with the NetCDF and CF community:

• Agree on a coordination mechanism to drive forward the NetCDF format (e.g. enhanced packing/compression) and the CF metadata standards to be fit for operational meteorology (WMO requirement)

• Consider the resources (e.g. at NMHS level) required to support the data format and metadata standard development

• Develop a model for the governance


Since the ET-ADRS Initial Meeting


Feedback on Meeting Report

• Suggested change to netCDF recommendation:

• “…WMO should look to adopt the netCDF4 API …but initially with the netCDF-4 classic model 'virtual format' (a combination of a real binary format and data model), as this will offer a high degree of interoperability, whilst addressing many of the shortcomings of netCDF3.”

• Raised concerns over the interpretation of phrase:

• “…important do distinguish between recommending it as a format and recommending the use of the API from Unidata.”

• Proposed various changes / additions to the Annex on the comparison of DRSs


Further progress

…it’s all gone rather quiet


Some Thoughts


Déjà vu?

• Meeting of Expert Team on the Development of GRIB2, Silver Springs, 2-5 Dec 1997

• Included Russ Rew (Unidata) & Mike Folk (NCSA) to present on netCDF & HDF

• Recommended:• Develop Roadmap for merge project with netCDF, HDF

and GRIB

• WMO to write to netCDF and HDF authorities expressing interest in closer collaboration and the potential for a standard common API

• Noted:• GRIB, netCDF and HDF complementary in many ways

• NetCDF and HDF could adopt GRIB2 as a basic physical format


Déjà vu? (2)

…and then it all went rather quiet


Changing Times?

• NetCDF now in very wide use:

• High profile – IPCC climate data

• Bastion of GRIB – ECMWF for ensemble data

• Likely that WMO CBS will provide some kind of endorsement of netCDF

• Will be driven forward by NMHS usage anyway

• E.g. Met Office looking to move away from bespoke format to CF-netCDF, GRIB-2 and BUFR


Opportunities / Threats

WMO involvement with CF…

• Opportunity to get additional resources allocated from WMO member states to develop / support the standard

• How will a slow-moving standards organisation like WMO work with a (relatively) quickly evolving standard like CF?

• Threat to the standard / process – currently both responsive to user requirements and very well thought-out, due to the fairly small, dedicated community involved


Questions and answersDiscussion