CF Metadata Conventions:
Governance, Support, and Future
Karl E. Taylor
Program for Climate Model Diagnosis and IntercomparisonLawrence Livermore National Laboratory
Presented to theGO-ESSP Community Workshop
Paris, France
11 June 2007
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Intro to CF metadata conventions
• Climate and Weather Forecast Metadata that makes data files self-describing
• Examples of CF metadata
Coordinate information needed to locate data in space and time
Further grid information (e.g., grid cell bounds, area)
Standard names – in conjunction with other attributes helps users determine whether data from different sources are the same physical quantity and helps distinguish variables within archives
Some processing info. (e.g., zonal mean, climatological processing)
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Brief history of CF
• Started with “COARDS” standard, and developed through efforts of a handful of volunteers
• 2003: CF 1.0 released
• 2005: CF white paper discussing future governance circulated
• 2006: White paper revised and presented to WCRP WGCM
• 2006: Current governance structure established
• 2003-present: increasing acceptance of CF and adoption by several community coordinated projects:
IPCC AR4 archive conforms
PMIP, HTAP, regoinal groups, seasonal groups, ENSEMBLES …
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Governance structure
• Original CF “authors” turned over control to two working committees:
Conventions Committee
Standard Name Committee
• Committee work done via email and web discussion (“Trac”)
• Support provided by:
BADC: Alison Pamment (50%) – standard names
PCMDI: Kyle Halliday (20%) – web site support
• WCRP/WGCM has been approached to appoint a CF Governance Panel charged with responsibility for stewardship of CF.
PCMDI BASC / CRC17 May 2007 K. E. Taylor
What happened at last GO-ESSP meeting?
• Discussed CF governance & procedure for making changes
• Some pressing CF issues were identified and in some cases individuals volunteered to try to make progress
Augmented grid information, especially for less structured grids.
Relationship between CF and GIS grids & metadata
Standard name issues (including dialects, profiles, hierarchy structures)
Relationship between CF and netCDF4
CF handling of in situ observations
How to handle discovery information
Sample datasets and reference implementations
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Agenda for today
• Governance issues – what needs fixing?
• Underlying principles – any adjustments needed?
• Technical issues – what’s been done? Can we make progress on any of the discussion topics?
• Supportive software – status reports
Any additions? Other concerns?
Conventions Committee:
Support and Governance
Agenda Item
GO-ESSP Community Workshop
Paris, France
11 June 2007
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Current committee members
• Karl Taylor (chair)
• Kyle Halliday (secretary)
• Balaji
• John Caron
• Jonathan Gregory
• Tom Gross
• Steve Hankin
• Jamie Kettleborough
• Russ Rew
• Rich Signell
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Terms of reference
• Further develop (revise) CF conventions
• Consider implementation of CF metadata conventions in other file formats (besides netCDF)
• Determine what is meant by CF conformance
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Current procedure for modifying/extending CF
• Anyone can propose a change
• Discussion opens & a member of the committee volunteers to moderate
• A provisional resolution is reached by consensus and summarized by moderator
• Reference files are produced illustrating the new feature
• If not a trivial change, trial implementations are carried out.
• If problems emerge iterate
• When consensus reached, the proposed change is provisionally accepted. (advanced implementers may begin adoption)
• Collection of changes evaluated as a package leading to new CF versions released by consensus
• Test reference files become part of the “test data”
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Problems encountered.
• Progress on some proposals, but few have been carried through to implementation.
• Contrast with standard names committee
Few actions have taken place
With a couple of exceptions, the volunteer moderation of discussion topics has floundered
• Procedure for introducing changes outlined in white paper:
Web-based Trac system
• Jonathan’s thoughts
Issues Concerning
Underlying Principles of CF
Agenda Item
GO-ESSP Community Workshop
Paris, France
11 June 2007
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Underlying principles
• Special needs of observations
• Should the scope of metadata accommodated by CF include:
Description of experiment Description of model Description of higher-order grid-characteristics (e.g., relationship between u & v grids,
which grid cells are connected to which)
• Competing needs of data producers & application developers
CF extensions can be implemented faster by producers who sometimes are impatient for changes to be formally adopted
• Should requirement be relaxed that all CF metadata reside in the same file as the data itself?
• Bundling of common definitions of a quantity into I.D.’s (defined by a combination of attributes including the standard name + others)
• Aggregation layers
• CF profiles
Miscellaneous Technical Issues
Agenda Item
GO-ESSP Community Workshop
Paris, France
11 June 2007
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Miscellaneous technical issues
• Ensembles (including realization_weight and other issues): Alison
• Time axis “issues” for forecasts: Jonathan
• Definition of the “geoid”: Alison
• Use of “anomaly”: Alison or Jonathan
• Subgrid variation description issues (use of “where”): Jonathan
• Time and calendar issues for paleoclimate simulations: Jean-Yves? Balaji?
The Future of CMOR
Karl E. Taylor
Program for Climate Model Diagnosis and IntercomparisonLawrence Livermore National Laboratory
Presented to theGO-ESSP Community Workshop
Paris, France
11 June 2007
PCMDI BASC / CRC17 May 2007 K. E. Taylor
The Climate Model Output Rewriter (CMOR)
• Provides a uniform interface for contributing CF-compliant data (adhering to “good practices”) to the growing number of MIP’s (e.g., CMIP, AMIP, CFMIP, PMIP, APE, HTAP)
• For IPCC AR4, nearly all model output in the archive was written through CMOR.
• Assures CF-compliance and all MIP-established output requirements are met.
• Traps many metadata errors when files are produced.
See http://www-pcmdi.llnl.gov/software/about_software.php
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Project-specific CMOR tables facilitate and ensure consistency of model output from contributing groups
• Proper specification of several coordinate attributes, including:
Correct standard name
“axis”, “positive”, and “formula_terms” attributes, as appropriate
• Proper specification of several variable attributes, including:
Correct standard name
Required dimensions
“cell_methods” attribute
• A capability to
Reorder axis order
Reverse axis direction (or translate longitude dimension)
Convert units (through udunits)
PCMDI BASC / CRC17 May 2007 K. E. Taylor
CMOR flagged common errors, including
• Pointing out when required metadata are omitted.
• Rejecting incorrect metadata (wrong units, inadmissible attribute values, etc.)
• Rejecting inconsistent coordinate dimensions passed by user to CMOR.
• Rejecting non-monotonic coordinate values or inconsistent boundary values, as passed by user.
• Rejecting values that are clearly unrealistic (likely indicating improper units conversion or incorrect sign).
PCMDI BASC / CRC17 May 2007 K. E. Taylor
What changes will be implemented in the next 6 months?
• Station data (needed, for example, by the HTAP project), including
Station names
Station locations
• Output from regional models
With logically rectangular grids
But with non-cartesian longitudes and latitudes
i.e., data(i,j), lon(i.j), and lat(i,j)
PCMDI BASC / CRC17 May 2007 K. E. Taylor
Longer term prospects for CMOR
• Difficult to see extending CMOR to write the grid-description files mandated by the GridSpec proposal.
• Probably possible, however, to write the actual data files through CMOR (and leave it to other software to produce the special GridSpec files).