Distribution of Real-Time and Retrospective NCEP

Model Data Sets

RT-NOMADS

Jordan C. Alpert jordan.alpert@noaa.gov

Jun Wang jun.wang@naa.gov Glenn Rutledge glenn.rutledge@noaa.gov Alpert 9/5/03

where the nation’s climate and weather services begin

NOAA Operational Model Archive and Distribution System

NOMADS is an agreement between agencies who participate to have common … Data and observation distribution software, format Data and observation distribution software, format

independent and description methodology,independent and description methodology, Documentation and organization framework,Documentation and organization framework, Forum to plan and organize Forum to plan and organize Funding avenue to direct $.Funding avenue to direct $.

… … for university, federal agencies and organizations, a for university, federal agencies and organizations, a mechanism to obtain support for the dissemination mechanism to obtain support for the dissemination of their data sets.of their data sets.

ParticipantsU.S.-NOAA, DOE, NASA,NCAR, Unidata, COLA,OPeNDAP Project, & Several Universities

International- CEOSNERCESGUK BADC BOM / Dept. of State via CAP

NOMADS and the NCEP Real Time Component

Wider information dissemination through web pages and Wider information dissemination through web pages and other avenues.other avenues.

Framework to create specialized and value added products Framework to create specialized and value added products needs access to weather data. needs access to weather data.

Facilitate communication among commercial, public, Facilitate communication among commercial, public, research community and federal agencies.research community and federal agencies.

NOMADS is a system for participants to interact with NOMADS is a system for participants to interact with others using common protocols and formats. Existing others using common protocols and formats. Existing differences in details are mollified by software. differences in details are mollified by software.

A solution for Recommendation 5 of “Fair Weather: Effective Partnership in Weather and Climate Services”

RT-NOMADS at NCEP

Distributed Ocean Data Server - OPEN_DAP (DODS) Distributed Ocean Data Server - OPEN_DAP (DODS) based on metadata file descriptors and XML. based on metadata file descriptors and XML.

Web based ftp, the NCEP Web based ftp, the NCEP ftp2uftp2u client and “Great client and “Great Displays” (pdisp), a grid analysis and display client.Displays” (pdisp), a grid analysis and display client.

Model initialization ingredients, “The minimum set” Model initialization ingredients, “The minimum set” for model rerun/retrospective analysis ready for model rerun/retrospective analysis ready observations & restart files.observations & restart files.

The Real Time NCEP Operational model run history.The Real Time NCEP Operational model run history.

Consists of

RT-NOMADS is a service for science, development and public access.

To distribute NCEP’s Operational data set to To distribute NCEP’s Operational data set to researchers and the public. researchers and the public.

To provide server software to NOAA projects who To provide server software to NOAA projects who need to serve their data through web services.need to serve their data through web services.

NCEP NOMADS servers are the intermediary NCEP NOMADS servers are the intermediary between NCEP operational computers and NCDC between NCEP operational computers and NCDC NOMADS archive servers.NOMADS archive servers.

Tools for Users on the server

Pare down large file sizes of high resolution Pare down large file sizes of high resolution initialization and model results. initialization and model results.

(re-) Group different data sets to create (re-) Group different data sets to create needed products – such as initialization files needed products – such as initialization files for model development.for model development.

Subset the data sets in parameter spaceSubset the data sets in parameter space Subset the data sets in physical space Subset the data sets in physical space (subset the grid in 4 dimensions)(subset the grid in 4 dimensions)

NCEP Real-Time NOMADS Server Real time.Real time. Operational Suite of Model products.Operational Suite of Model products. Forecast information for model comparison.Forecast information for model comparison. Operational initialization and observations.Operational initialization and observations. Maintenance of codes. Maintenance of codes. Utilities for data manipulation and grid transformation & sub-setting. Utilities for data manipulation and grid transformation & sub-setting. Documentation.Documentation. Serves as transfer point to NCDC for NOMADS archive of reruns Serves as transfer point to NCDC for NOMADS archive of reruns

and retrospectives.and retrospectives.

The Server – Client distributed relationship:A Common Sense Approach to Data Distribution

ftp2uftp2u, Great Displays, and GDS allow sub-setting , Great Displays, and GDS allow sub-setting of data by parameter, by 3D region, and by time.of data by parameter, by 3D region, and by time.

Allows for results with low band width! Eg., Allows for results with low band width! Eg., NASA/GSFC takes 1 hour to ftp entire GFS NASA/GSFC takes 1 hour to ftp entire GFS (AVN) forecasts or 13 seconds to download their (AVN) forecasts or 13 seconds to download their desired fields (desired fields (ftp2uftp2u).).

GDS: NOMADS participants serve up their data GDS: NOMADS participants serve up their data sets through a client-server relationship, that is, sets through a client-server relationship, that is, the data sets are internet ready and the display is the data sets are internet ready and the display is done by the client. done by the client.

Examples of Clients :

Extending NOMADS to Real Time The A in NOMADS is for Archive.The A in NOMADS is for Archive. NCEP historical cases for rerun/retrospective case studies NCEP historical cases for rerun/retrospective case studies

of ocean, atmosphere and climate.of ocean, atmosphere and climate. NCEP Real Time model output in the NOMADS NCEP Real Time model output in the NOMADS

framework … framework … so there is a seamless accessso there is a seamless access to these to these important data sets at NCEP and at NCDC! important data sets at NCEP and at NCDC!

Use of benchmarks to characterize system loads in order Use of benchmarks to characterize system loads in order to determine costs and give NOAA administration the to determine costs and give NOAA administration the information it needs to triage data resources. information it needs to triage data resources.

RT-NOMADS provides the framework for long term support RT-NOMADS provides the framework for long term support to be available to keep up with modeling advances.to be available to keep up with modeling advances.

RT-NOMADS is a prototype

Provide proof of the concept that distributed systems Provide proof of the concept that distributed systems can serve data sets across agencies for modeling can serve data sets across agencies for modeling science, the university community and . science, the university community and .

Specify an Operational system for NOAA/TOC,Specify an Operational system for NOAA/TOC, Provide current state of the art operational capability Provide current state of the art operational capability

for serving observations and data to Forecast Offices, for serving observations and data to Forecast Offices,

Goals:

Data Application Protocol (DAP) using…

internetDODS requests and

compressed data exchangedvia HTTP

GRIB dataNetCDF data

binary data

HDF data

datasets in any formatsupported by GrADS

GrADSbatch mode

interfacecode

DODS serverlibraries

GrADS-DODS Serverextracts meta-

data and subsetsmaps DODS requests

to GrADS services

Javaservlet

parses requests,packages data

handles HTTPprotocol

DODS client libraries

GrADS

Matlab

IDL

etc..

data appears to client as local file, in astandard format (i.e, NetCDF, etc.)

Client

handles HTTP, unpacks data

etc..

… GrADS client …. commands

Sdfopen http://nomad2:9090/dods/reanalysis-2/pgb/hgtSdfopen http://nomad2:9090/dods/reanalysis-2/pgb/hgt

t = 1t = 1

'set gxout shaded''set gxout shaded'

while (t <= 1068)while (t <= 1068)

'set t 't'set t 't

'q time''q time'

date=subwrd(result,3)date=subwrd(result,3)

'draw title hgtprs 'date' 't'draw title hgtprs 'date' 't

'd hgtprs''d hgtprs'

'/home/wd23ja/grads/lib/cbarn.gs''/home/wd23ja/grads/lib/cbarn.gs'

'swap''swap'

t = t + 1t = t + 1

endwhileendwhile

Getting data without GrADS or a commercial client:

http://140.90.198.156:9090/dods/eta/eta20030527/eta_00z.ascii?http://140.90.198.156:9090/dods/eta/eta20030527/eta_00z.ascii?tmpprs[0:0][0:0][30:130] [290:290]tmpprs[0:0][0:0][30:130] [290:290]

#the order of the square bracked values is:#the order of the square bracked values is:

# [time][level][lat][lon]# [time][level][lat][lon]

# where lat is measured from the south pole (0)# where lat is measured from the south pole (0)

# to equator (90) to NP (180)# to equator (90) to NP (180)

(Use a “wget” and the URL with the ascii response in cron or a cgi-bin (Use a “wget” and the URL with the ascii response in cron or a cgi-bin script provides the needed number(s) when and where needed.) script provides the needed number(s) when and where needed.)

NOMADS clients at NCEP

Real Time Operational model data sets from GFS, and Real Time Operational model data sets from GFS, and ETA from present to one month ago. ETA from present to one month ago.

Attention to accumulation variable differences and high Attention to accumulation variable differences and high and low resolution.and low resolution.

Analysis pgb, restart files.Analysis pgb, restart files. Recent (“real time”) climate runs, ie., RSM. Recent (“real time”) climate runs, ie., RSM. Observations (soon).Observations (soon). Reanalysis-2 (and 1)Reanalysis-2 (and 1) AMIP type runs (Suru’s finest runs eg., 28 and 64 levels AMIP type runs (Suru’s finest runs eg., 28 and 64 levels

1978-2002), sudden warming, ….1978-2002), sudden warming, ….

“Great Displays” (pdisp) and ftp2u by Ebisuzaki and modified by Wang

MATLAB: A Commerical Client

server=‘nomad2:9090’;server=‘nomad2:9090’;directory=‘/dods/gdas/rotating/’; fext=‘ascii’;directory=‘/dods/gdas/rotating/’; fext=‘ascii’;model=‘gdas’; datestr=‘YYYYMMDDHH’; varname=‘hgt’;model=‘gdas’; datestr=‘YYYYMMDDHH’; varname=‘hgt’;varlevs=‘1’; varlevs=‘1’; Arrc=[‘[‘,t1,’:’,t2,’][‘,nlev1,’:’,nlev2,’][‘,ilat1,’:’,ilat2,’][‘,ilon1,’:’,ilon2,’]’];Arrc=[‘[‘,t1,’:’,t2,’][‘,nlev1,’:’,nlev2,’][‘,ilat1,’:’,ilat2,’][‘,ilon1,’:’,ilon2,’]’];urlstring=[http://’,server,directory,model,datestr,’.’,fext,’?’,varname,arrc];urlstring=[http://’,server,directory,model,datestr,’.’,fext,’?’,varname,arrc];url = java.net.URL(urlstring); % connect to web pageurl = java.net.URL(urlstring); % connect to web pageTryTry stream = openStream(url)stream = openStream(url) ireader = java.io.InputStreamReader(stream);ireader = java.io.InputStreamReader(stream); breader = java.io.BufferedReader(ireader);breader = java.io.BufferedReader(ireader);Line = readLine(breader); disp(line); Line = readLine(breader); disp(line);

NCEP Data Sets

NOAAPORT Operations.NOAAPORT Operations. Continuation of NCEP real time model Continuation of NCEP real time model

component.component. NCEP model initialization ingredients, NCEP model initialization ingredients,

“The minimum set” for model “The minimum set” for model rerun/retrospective analysis ready rerun/retrospective analysis ready observations & restart files.observations & restart files.

Archives at NCDC

Model Data Requirements from CLASS

Current Archive:NOAAPort: 2TB/yrGDAS/SSI: 400Gb/yrNARR 10TbTotal 12.5TbFY04NCEP Hi-Res: +10TB/yr*Total 22.5Tb

* 5 year retention only

North American Regional Reanalysis

NCEPGlobal AssimilationSpectral Statistical

GDAS O/I

NCEP Hi-ResEta/Globaland NARR

NOAAPortGridded Output

HDSS23TB/yr

LegendGreen- Existing ArchiveBlue- Expected FY03/04

Rutledge and Alpert 6/03

FY03

Model Data Req’s - NWP and GCM’s

0

20

40

60

80

100

120

140

'04 '05 '06 '07

Volume Tb/Yr

GCMReanalNWP*

Cumulative (Tb)

0

50

100

150

200

250

'04 '06 '08 '10

NWP*ReanalGCM

GCM- Global Climate ModelReanal- N. Am. Regional ReanalysisNWP- Numerical Weather Prediction

* 5-year rotating NWP archive

Alpert & Rutledge 6/03

RT NOMADS Server at NCEP

Two 1TB RAIDZONE serversTwo 1TB RAIDZONE servers http://nomad2.ncep.noaa.govhttp://nomad2.ncep.noaa.gov http://nomad1.ncep.noaa.govhttp://nomad1.ncep.noaa.gov (backup) (backup) Documentation web page under constructionDocumentation web page under construction New disk space on order….New disk space on order….

Non-Operational “Projects” can become NOMADS participants to display their data sets:

Install scripts and clients, Install scripts and clients, ftp2uftp2u and “Great and “Great Displays” (pdisp) for a GDS framework,Displays” (pdisp) for a GDS framework,

Setup GDS XML file – see example on web Setup GDS XML file – see example on web page, for example,…page, for example,…

Including GDS administration with updates Including GDS administration with updates and new additions,and new additions,

““newxml” update scripts – to automatically newxml” update scripts – to automatically add new directories and files.add new directories and files.

NOMADS gives guidance to:

(RT) NOAMDS data sets ….

One data set (e.g, GRIB) provides the data One data set (e.g, GRIB) provides the data for all access applications.for all access applications.

GraDS control files (grib2ctl) and index GraDS control files (grib2ctl) and index files (gribmap) create unique metadata files (gribmap) create unique metadata description and random access.description and random access.

GDS can read HDF, NETCDF* as well as GDS can read HDF, NETCDF* as well as other formats.other formats.

NOMADS and the NCEP Real Time Component

NWS should make its data and products available in NWS should make its data and products available in Internet-accessible digital form.Internet-accessible digital form.

Information held in digital databases should be based on Information held in digital databases should be based on widely recognized standards, formats, and metadata widely recognized standards, formats, and metadata descriptions to ensure that data from different observing descriptions to ensure that data from different observing platforms, databases and models can be integrated and platforms, databases and models can be integrated and used by all interested parties in the weather and climate used by all interested parties in the weather and climate enterprise.enterprise.

A solution for Recommendation 5 of “Fair Weather: Effective Partnership in Weather and Climate Services”