The NCAR Command Language (NCL) and the NetCDF Data Format

Research Tools PresentationMatthew Janiga

10/30/2012

Where Can I Learn More?

http://www.ncl.ucar.edu/index.shtml

Where Can I Learn More?

http://www.unidata.ucar.edu/software/netcdf/

What is NCL?• NCL is a complete language with types, variables, operators,

expressions, conditional statements, loops, and functions and procedures.

• NCL can read many data formats: NetCDF, HDF, GRIB, ascii and csv, binary, WRF output, and more.– With some additional work it can read GEMPAK, McIDAS

AREA, and others.• NCL like MATLAB is an array based language. Arrays can be

sent to functions (100s and growing) which perform computations, plotting functions (dozens and growing), or files (typically NetCDF but also ascii and binary).

What is NetCDF?• NetCDF is a data format for storing arrays. Unlike ascii it also stores information

describing the arrays.– Attributes: strings or numbers storing: units, source locations, references,

methodology, etc….– Dimension coordinates: Arrays which describe the dimensions: lat, lon, time,

level, etc. Dimensions coordinates can have their own attributes.– NetCDF files are EASILY examined using GUIs (NCVIEW, IDV, Panopoly, etc.) or

command line programs (ncdump, ncl_filedump).– NetCDF files can also be created, read, or manipulated by functions in libraries

written for Fortran, C, Java, and more!• How difficult is say… compiling a Fortran program that uses NetCDF?

– Just type "nc-config --libs“ on the command line and all flags will be given to you!

• NetCDF is also fairly easy to read into programs other than NCL: R, MATLAB, etc.– http://www.r-project.org/

NetCDF JavaThe NetCDF Java is a jar file opened from the web that can convert many

formats to NetCDF.http://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp

http://nco.sourceforge.net/

NetCDF OperatorAn efficient command line program for merging, splicing, and otherwise editing NetCDF files. I’ve found NCL to be much easier to use and powerful but this is another option.

NCVIEW

NCVIEW is a simple GUI that lets you explore a NetCDF file: examine slices, time series, animate, and more. Look at the data you’ve generated before spending time to make a plot!

http://www.unidata.ucar.edu/software/netcdf/software.html

NCVIEW is one of MANY programs that do this.

Advantages of NCL• Free, stable, and extremely well documented.• NCL is a fully fledged programming language not calls to compiled programs.

Code yourself don’t “trick”.• NCL allows you to read in large amounts of data into memory (64-bit vars >

2GB) and perform computations there instead of constantly reading and writing to disk.

• Creating your own procedures to call externally is exactly the same as writing a script (no Fortran or shell scripting necessary).

• Handles many vertical and horizontal projections and conversions from one to the other.

• NCL is very flexible, it can analyze any array of any dimension, type, with any coordinate description.– Store and process: trajectories, tracks, hovmollers, time series, polar orbit

swaths and other non-uniform projections, radar data, geostationary satellite data, surface obs, profiler data, radiosondes, and pretty much anything you can think of.

Disadvantages of NCL• NCL has a bit of a learning curve. Learning how to fully utilize it and

code using arrays takes time.• For some plots (plan plots and cross-sections) GEMPAK may be able to

produce a similar plot with less code (assuming data is already in GEMPAK format).

• GEMPAK is still heavily used in operations.• While there are lots of mathematical functions and procedures

MATLAB has more types and variants.• Similarly R has more statistical tools.

NCL Eye Candy

And more…..

What’s Else can NCL Do?• OPeNDAP: The ability to grab what you

want out of datasets directly from a server. Makes some local storage of data obsolete.– Available for many datasets and

reanalyses.• system and systemfunc

– avail_files = systemfunc("ls -1")– time = systemfunc("date +'%s'")– system(“convert img.ps img.png &”)

• Use WRAPIT to send information to and from pre-compiled user-created or commercial Fortran or C programs– http://www.ncl.ucar.edu/Document/To

ols/WRAPIT.shtmlhttp://www.ncl.ucar.edu/Applications/editor.shtml

NCL for EMACS

More Features…

• ParNCL - Parallel version of NCL for analyzing large datasets.

• Earth System Modeling Framework (ESMF) regridding software.

• NetCDF 4 – Ability to store and read groups “object-orientated variables”.

• MJO CLIVAR Diagnostics – MATLAB still more flexible.

The Example Page

http://www.ncl.ucar.edu/Applications/

Documentation! The NCL webpage has 100s of examples each containing a picture of the plot and the code required to make it.

Function Documentation

Every function has an exhaustive documentation page showing how to use it, describing how the calculation is performed, and providing examples.

The Example Code

Libraries: NCLs and your own.

Read data

Perform the calculation

A Simple Plot

Code to make this plot