View
3
Download
0
Category
Preview:
Citation preview
Slide 1
© ECMWF February 23, 2016
MARS – Introduction and basic conceptsComputer User Training Course 2016
Carsten MaassUser Supportadvisory@ecmwf.int
COM INTRO 2016: MARS Introduction and basic concepts Slide 1
Slide 2
Introduction
Meteorological content
MARS language
MARS architecture
Retrieving data
Practicals
Contents
COM INTRO 2016: MARS Introduction and basic concepts 2
Slide 3
Introduction
COM INTRO 2016: MARS Introduction and basic concepts 3
Meteorological Archival and Retrieval System
Meteorological data (GRIB: fields, BUFR, ODB: observations) Large amount of data (size of archive & number of fields) Operational & Research environment Batch & interactive modes Large number of users with different requirements:
large datasets rarely ↔ few fields very often Heterogeneous environment
Slide 4
Client/Server architecture Clients: workstations, supercomputers Servers: supercomputers, dedicated servers Several databases Tape library
Introduction – MARS components
COM INTRO 2016: MARS Introduction and basic concepts 4
Slide 5
94 PiB of data in ~ 230 billion (2.3 · 109) meteorological fields in 12 million files
~ 600 GiB of metadata 150 · 106 fields added daily (100 TB) 1000 active users/day executing ~ 2 million requests/day ~ 100 TB retrieved daily Operational forecast since 1985 Analysis, forecast and observations since 1900 (ERA-20C)
Introduction – Some figures
COM INTRO 2016: MARS Introduction and basic concepts 5
Slide 6
Medium-range the high-resolution and the ensemble forecasts of weather, at the space and time-scales
represented by the relevant model, up to 10 and 15 days ahead, respectively, and the
associated uncertainty
Extended-range (monthly)ensembles of individual forecasts and post-processed products of average conditions (e.g.
weekly averages) up to 46 days ahead, and the associated uncertainty
Long-range (SEAS)ensembles of individual forecasts and post-processed products of average conditions (e.g.
monthly averages) up to 13 months ahead, and the associated uncertainty
COM INTRO 2016: MARS Introduction and basic concepts 6
Terminology – Forecast lead times
Slide 7
Re-forecast forecasts run for past decades necessary to estimate the model climate
and the level of skill and to generate some of the operational products
IFS
`Integrated Forecasting System`, the system used at ECMWF
Terminology – … more
COM INTRO 2016: MARS Introduction and basic concepts 7
www.ecmwf.int/en/what-naming-convention-ecmwf-real-time-products
www.ecmwf.int/en/forecasts/documentation-and-support
Slide 8
4DVAR (T1279 / 16km, T255 inner loops, input to HRES) - At synoptic hours 00, 06, 12 and 18 UTC- Surface - Model levels (137)- Pressure levels (25)- Isentropic levels (15 PT, 1 PV)
EDA (T399 / 50 km, T159 inner loops, input to ENS)- At synoptic hours 00, 06, 12 and 18 UTC- 26 members- Surface - Model levels (137)- Pressure levels (25)- Isentropic levels (16 PT, 1 PV)
Meteorological content – Operational Analyses
COM INTRO 2016: MARS Introduction and basic concepts 8
Slide 9
Atmospheric Forecast (10 day forecast based on 00/12 UTC Analysis) at T1279L137 (16 km)
- Surface- Model levels (137)- Pressure levels (25)- Isentropic levels (16 PT, 1 PV) - 1 hourly steps from 0 to 90, 3 hourly from 93 to 144 and 6 hourly from 150 to 240
hours
Meteorological content – HRES
COM INTRO 2016: MARS Introduction and basic concepts 9
Slide 10
Meteorological content – ENS Medium-range forecasts to 15 days, 91 Levels
26 member Ensemble of Data Assimilations (EDA, stream elda)
1 control forecast (as HRES but with lower resolution)
50 different forecasts with perturbed initial conditions
Truncation at day 10 from T639 (~32 km) to T319 (~64 km)
Two additional calibration/validation runs were added
Leg 3: 00 UTC FC extended Mondays & Thursdays to day 46
www.ecmwf.int/en/forecasts/documentation-and-support/extended-range-forecasts
# FC Leg 1 day 0-10
Leg 2 day 11-15
VAR-CF 1 T399 T255VAR-PF 50 T399 T255CV-HRES 1 T399CV-LRES 1 T255T2551CV-T319
T3991CV-T639T255T39950ENS-PFT255T3991ENS-CF
Leg 3day 16-46
Leg 2 day 11-15
Leg 1 day 0-10# FC
T3191CV-T6391CV-
T319T63950-PFT319T6391-CF
Leg 3day 16-
Leg 2 day 11-15
Leg 1 day 0-10# FC
COM INTRO 2016: MARS Introduction and basic concepts 10
Slide 11
Control forecast Calibration/Validation forecasts 50 perturbed forecasts / 26 EDA members Initial condition perturbations Ensemble mean and standard deviation Extreme forecast index Event probabilities Cluster mean, cluster representatives and standard deviation Trajectories (of tropical cyclones)
Meteorological content – ENS products
COM INTRO 2016: MARS Introduction and basic concepts 11
Slide 12
Forecast/Analysis Domain Number of
membersHorizontal resolution
Numberof
directions
Numberof frequencies
SAW HRES
Analysis +forecast
0–10 days Global 1 11 km 36 36
WAM HRES
Analysis +forecast
0–10 days Global 1 28 km 36 36
WAMENS
Forecast0–10 days Global 51 55 km 24 30
WAMENS
Forecast11–32 days Global 51 55 km 12 25
WAM SEAS
Forecast0–13months Global 51 111 km 12 25
Meteorological content – Ocean-Wave component
COM INTRO 2016: MARS Introduction and basic concepts 12
Slide 13
HRES forecast (Short cut-off forecast T1279L137 at 06/18) Analysis (4DVAR) Forecast (to 90 hours) in hourly steps 00/12 UTC AN/FC is taken from HRES
Additional ENS at 06/18 UTC available since 8 July 2015 3-hourly steps out to forecast range 144 hours
Meteorological content – Boundary-Condition Programme (BC)
Real-time data only available for participating MS
COM INTRO 2016: MARS Introduction and basic concepts 13
Slide 14
Grid resolutio
n (km)
HRES
ENS ENS Extended
4DVarLoops
EDALoops
HRES-WAM
ENS-WAM ENS-WAM Extended HRES-
SAWDay
0 - 10Day
11 – 15Day
16 - 46 1st 2nd 3rd Outer 1st 2nd Day 0 - 10
Day 11 - 15
Day 16 - 46
128 TL159 TL159
100 TL191 TL191
80 TL255 TL255 TL255
64 TL319 TL319 TL319
55 0.5° 0.5° 0.5°
50 TL399 TL399
36 TCO319
32 TL639
28 0.25° 0.25° 0.25°
18 TCO639 TCO639 TCO639
16 TL1279
14 0.125°
11 0.1°9 TCO1279
MARS – IFS cycle 41r2 resolution changes
COM INTRO 2016: MARS Introduction and basic concepts 14
https://software.ecmwf.int/wiki/display/FCST/Planned+changes+to+the+forecasting+system
Slide 15
SEAS – atmosphere-ocean coupled model (51 members)
Global forecasts from 00 UTC to 7 months: (once a month)- atmosphere: ~75 km resolution, 91 levels (T255 L91)
- ocean: NEMO – ORCA1 grid (~1˚x1˚ with equatorial refinement), 42 levels
In February, May, August and November, 15 of the 51members are extended to 13 months
Re-forecasts: 15 members (0-13m) covering 30 years (1981-2010) Part of the EUROSIP system, with UK Met Office, Météo France and
NCEP Availability of products: 12:00 on the 8th of each month
Meteorological content – Seasonal System 4
http://www.ecmwf.int/en/forecasts/documentation-and-support/long-range
COM INTRO 2016: MARS Introduction and basic concepts 15
Slide 16
Averaged over each calendar month
Atmosphere / Wave - Analysis
Surface / pressure levels
Meteorological content – Monthly Means
COM INTRO 2016: MARS Introduction and basic concepts 16
Slide 17
Projects
- DEMETER: Multimodel Ensemble for seasonal to Interannual prediction- Data targeting system- ENSEMBLES- EURO4M- MACC- PROVOST- ECSN-Hyretics- …
Meteorological content – Special datasets (1/2)
COM INTRO 2016: MARS Introduction and basic concepts 17
https://software.ecmwf.int/wiki/display/UDOC/MARS+content#MARScontent-Specialdatasets
Slide 18
IFS Research experiments- ECMWF
- Member States
Member States’ Projects- COSMO-LEPS - Aladin-LEAF- …
Meteorological content – Special datasets (2/2)
COM INTRO 2016: MARS Introduction and basic concepts 18
Slide 19
Meteorological content – Reanalysis datasets
COM INTRO 2016: MARS Introduction and basic concepts 19
Slide 20
38 years (1/1979 – 12/2016) of validated ERA-Interim analysis products are available
Continued in near real-time (with ~2 months delay)
Uses IFS Cycle 31r2, and 12h-4DVar
Resolution:
- Horizontal: T255, N128 (~0.7º)
- Vertical: 60 ML, 37 PL, 16 PT, PV=±2
Analyses at 00/06/12/18, Forecasts at 00/12 to 240 h
Subset of products also publicly available on the ECMWF Data Server at fullresolution
Meteorological content – ERA-Interim
COM INTRO 2016: MARS Introduction and basic concepts 20
Slide 21
Meteorological content – ERA-CLIM reanalysis products
Atmospheric reanalysis for the 20th-century (1900-2010)Using an ensemble of 10 plausible SST/sea-ice evolutions
Assimilating observations of surface pressure and marine wind125/25 km global resolution, 91 vertical model levels
ERA-20CM
ERA-20C
ERA-20CL
Ensemble of model integrations(mainly monthly products)
IFS Cy38r1 + CMIP5 data +HadISST v2.1
Assimilation of surface observations(3-hourly products)
+ ICOADS v2.5.1 +ISPD v3.2.6 (incl. ERA-CLIM)
High-resolution land surface (25km global) + CHTESSEL
Final ERA-20C/M/L datasets (~200 Tb) are becoming available athttp://apps.ecmwf.int/datasets
COM INTRO 2016: MARS Introduction and basic concepts 21
Slide 22
ERA-20C: A terrific storm at sea3 February 1899
COM INTRO 2016: MARS Introduction and basic concepts 22
Slide 23
ERA-Interim ERA5Start of production August 2006
IFS Cy31r2January 2016IFS Cy41r2
Model input As in operations(inconsistent SST)
Appropriate for climate(CMIP5, HadISST.2)
Spatial resolution 79 km global60 levels to 10 Pa
31 km global 137 levels to 1 Pa
Time period 1979 - present 1979 - present
Dissemination Monthly Monthly for ERA5; daily for ERA5T
Observations Mostly ERA-40, GTS Various reprocessed CDRs
Radiative transfer RTTOV7 RTTOV11
Analysis method 4D-Var1D+4DVar rain
10-member ensemble 4D-Var (EDA)All-sky MW
Variational biascorrections
Satellite radiances Also ozone, aircraft, surface pressure
Meteorological content – Next ERA-Interim
COM INTRO 2016: MARS Introduction and basic concepts 23
Slide 24
Atmosphericreanalyses
2013 2014 2015 2016 2017 2018
ERA-InterimERA-Interim/Land
ERA-20CMERA-20CERA-20C/Land
ERA5ERA5/Land
CERA-20CCERA-20C/Carbon
20th-century reanalysis1900–2010, 125km resolution
ERA-Interim replacement
1979–NRT, 31km resolution
Coupled with ocean1900-2010, 125km/1x1
1979–NRT, 79km resolution
Meteorological content – Reanalysis outlook
COM INTRO 2016: MARS Introduction and basic concepts 24
Slide 25
Meteorological content – TIGGE
THORPEX Interactive Grand Global Ensemble Global ensemble forecasts to around 14 days generated routinely at
different centres around the worldECMWF, JMA (Japan), Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), CPTEC (Brazil), KMA (Korea)
Data archived in GRIB 2 TIGGE-LAM data since 1/1/2013
http://tigge.ecmwf.int
COM INTRO 2016: MARS Introduction and basic concepts 25
Slide 26
Sub-seasonal to seasonal prediction
Joint WWRP/THORPEX-WCRP research project established to improve forecast skill and understanding on the sub-seasonal (up) to seasonal time scale, and promote its uptake by operational centres and exploitation by the applications community
Following the TIGGE approach
Provides real-time + reforecasts
So far data from 9 centres, a further 2 interested
Meteorological content – S2S
COM INTRO 2016: MARS Introduction and basic concepts 26
https://software.ecmwf.int/wiki/display/S2S/Home
Slide 27
Observations- Surface data- Vertical soundings- Upper-air data- Satellite
Feedback Analysis Input Analysis Feedback (superseded by ODB feedback)
Meteorological content – Observations & Feedback
COM INTRO 2016: MARS Introduction and basic concepts 27
Slide 28
WMO formats Fields in GRIB (GRid In Binary), ECMWF local extensions
- Spherical Harmonics (upper-air fields, T1279)
- Gaussian Grid (surface fields, N640)
- Latitude/Longitude (wave and ocean products)
Observations in BUFR (Binary Universal Form Representation)- Instrument specific
ECMWF/IFS format ODB (Observational Data Base)
- Observation feedback
Meteorological content – Data formats
COM INTRO 2016: MARS Introduction and basic concepts 28
Slide 29
In the IFS observations are handled by ODB ODB is a
- Hierarchical in-core database with a data definition and query language: ODB/SQL
- A data format- . . .
ODB Observation Feedback (ofb) data is archived in MARS- Improve the representation of feedback data in MARS meta data
- Introduce SQL capabilities to request feedback data
To improve the handling of observations, ODB will be further integrated into ECMWF systems
ODB can be handled by Metview, see
software.ecmwf.int/wiki/display/METV/Tutorials
MARS – ODB
COM INTRO 2016: MARS Introduction and basic concepts 29
Slide 30
Content - Simulated satellite images (global on Gaussian grid)- ERA-CLIM2 (CERA-20C) - UERRA (Uncertainties in Ensembles of Regional Re-Analysis)- JRA-55- Copernicus datasets (ERA5, Multi-model Seasonal predictions,
Regional Reanalyses, … )
Architecture - New interpolation package- Alignment with new Product Generation
MARS – future development
COM INTRO 2016: MARS Introduction and basic concepts 30
Slide 31
Mechanism to name fields
Request syntax:verb,
keyword1 = value1,… = value2,keywordN = valueN
verb: action to be taken (e.g. retrieve, list, read) keyword: a known MARS variable, e.g. type or date value: value assigned to the keyword, e.g. Analysis or temperature
MARS language
COM INTRO 2016: MARS Introduction and basic concepts 31
Slide 32
verb and keyword=value separated by commas, but last one
Spaces and tab characters are ignored
*, ! and # comment until end-of-line
Directives are not case sensitive
Values: predefined names, numeric values or strings (filenames)
Abbreviations: enough letters to uniquely identify keyword or value
Acronyms: usually initial letters of names
/ is used as list separator → specify pathnames in quotes
MARS language
COM INTRO 2016: MARS Introduction and basic concepts 32
Slide 33
retrieve, actionclass = od, identificationstream = oper,expver = 1,
date = -3, date & time relatedtime = 12,
type = analysis, data relatedlevtype = model levels,levelist = 1/to/137,param = temperature,
grid = 2.5/2.5, post-processing
target = “analysis” storage
MARS language – Retrieve request
COM INTRO 2016: MARS Introduction and basic concepts 33
Slide 34
MARS language – Identification of archive
class ECMWF classification (od, rd, e4, …)
stream
expver
domain
origin
system
method
originating forecasting system or (oper, wave, enfo, seas, …)
version of the experiment (01 operational, 11, aaaa)
area covered by the data (Global, Mediterranean, …)
originating centre of the data (kwbc, egrr, …)
seasonal forecast operational system (1, 2, 3)
to specify how the seasonal forecast is produced, e.g. in System
2, method=0 for runs without ocean assimilation (0, 1, …, 3)
COM INTRO 2016: MARS Introduction and basic concepts 34
Slide 35
MARS language - Date & timetime base time or observation time (00, 06, 09:30, …)
date base date of the model (-1, 20010225, …)
step forecast time-step [hours] from base time (12, 24, 240, …)
reference reference forecast time step for EPS tube (96,…)
refdate date of real-time forecast associated to re-forecast/hindcast(stream=mnfh)
hdate base date of a re-forecast/hindcast (stream=enfh)
range observations: period in minutes from base time (360,…)ocean fields: extension of the time series/average
fcmonth month from seasonal forecast base date (1, 6, …)
fcperiod period, in days, for an averaged field (26-32)
COM INTRO 2016: MARS Introduction and basic concepts 35
Slide 36
MARS language – Fieldstype type of field (an, fc, …)
levtype type of level (pl, ml, sfc, pt, pv)
levelist levels for the specified levtype (off if levtype=sfc)
param meteorological parameter (t, temperature, 130, 30.128)
number ensemble member (1, 2, …)
channel brightness temperature frequency band
diagnostic, iteration sensitivity forecast products
frequency, direction 2-d wave spectra products
product, section, latitude, longitude ocean products
COM INTRO 2016: MARS Introduction and basic concepts 36
Slide 37
MARS language – Observations & images
type type of observations or images (ob, fb, ai, af, im)
obstype observation subtype (s, air) or image channel
ident WMO observation station number or satellite identifier
duplicates whether duplicated observations are to be kept or not
block WMO block number for observation
time analysis time (types ai, af) or observations time (types ob, fb, im)
range denotes the period, in minutes, starting from time
COM INTRO 2016: MARS Introduction and basic concepts 37
Slide 38
MARS language – ODB
type Type of ODB information, ofb (ODB Feedback), mfb (MONDB Feedback), oai (ODB Analysis Input)
reportype classification to index ODB data (16020)
obsgroup Grouping of report types (optional)
type Type of ODB information, ofb (ODB Feedback), mfb (MONDB Feedback), oai (ODB Analysis Input)
time time represents the analysis time (ODB column antime)
filter SQL filter query ("select lat,lon,obsvalue where varno=39“)
COM INTRO 2016: MARS Introduction and basic concepts 38
ODB Governance database: http://data-portal.ecmwf.int/odbgov/
Slide 39
Unix pathnames (using /) have to be enclosed in quotes, e.g.target = “/scratch/ms/gb/uid/analysis”
MARS language – Storage
target UNIX pathname where retrieved data is stored
source UNIX pathname from where to read data
fieldset temporary storage; can be considered a MARS variable
COM INTRO 2016: MARS Introduction and basic concepts 39
Slide 40
MARS language - Post-processing (1/2)grid output grid mesh
Latitude/longitude increments in degrees (2.5/2.5)
Type and resolution of Gaussian grid, e.g. • grid = F320 – full (or regular) Gaussian grid
• grid = N320 – ECMWF original reduced Gaussian grid (only selected resolutions supported)
• grid = O320 – ECMWF octahedral (reduced) Gaussian grid
All above with 320 latitude lines between the pole and equator
COM INTRO 2016: MARS Introduction and basic concepts 40
F80 regular (or full) N80 original reduced O80 octahedral reduced
Slide 41
MARS language - Post-processing (2/2)
area desired sub-area in degrees (north/west/south/east)
frame number of grid points from sub area inwards (5)
resol triangular truncation (319, auto, av)
rotation lat/lon of South Pole
accuracy number of bits per data value in GRIB (16)
style specify post-processing style (dissemination)
COM INTRO 2016: MARS Introduction and basic concepts 41
Slide 42
MARS language – Execution control
expect number of expected fields (1000, any, …)
database where to look for the data
use hint about frequency of use (infrequent)
COM INTRO 2016: MARS Introduction and basic concepts 42
Slide 43
Single value, predefined names, numbers, mnemonicsparam = temperature
List of values, separated by /step = 12/24/48
Range of values, using keywords: to, / and bydate = 20020101/to/20020131
step = 24/to/240/by/24
MARS language – Values
COM INTRO 2016: MARS Introduction and basic concepts 43
Slide 44
Expected number of fields is computed by multiplying number of values after expansion of ranges
date = 20020101/to/20020131 31 fields
Certain keywords accept all as valid valuelevelist = all
Most keywords accept off as valid valuelevtype = surface,
levelist = off
Not all possible combinations keyword = value name an archived field
MARS language – Values
COM INTRO 2016: MARS Introduction and basic concepts 44
Slide 45
Retrieval of snow depth from the ERA-Interim archive for December 2007, for all analysis base times. It retrieves 124 fields.
retrieve,class = ei,stream = oper,expver = 1,date = 20071201/to/20071231,time = 00/06/12/18,type = an,levtype = sfc,param = sd,target = “era-int.200712.sd”
Request examples – Interim Reanalysis
COM INTRO 2016: MARS Introduction and basic concepts 45
Slide 46
Retrieval of surface temperature and 10-m wind components (U and V), 20 first members of the EPS for 2nd Jan 2001 for time steps 12, 36 and 60. It retrieves 180 fields.
retrieve,class = od,stream = enfo,expver = 1,date = 20010102,time = 12,step = 12/36/60,type = pf,levtype = sfc,param = st/10u/10v,number = 1/to/20,target = “perturbed.sfc”
Request examples - Ensemble forecast
COM INTRO 2016: MARS Introduction and basic concepts 46
Slide 47
Retrieval of sea surface temperature for first 10 days of May 2002, all synoptic times. It retrieves 40 fields.
retrieve,class = od,stream = oper,expver = 1,date = 20020501/to/20020510,time = 00/06/12/18,type = an,levtype = sfc,param = sea surface temperature,target = “sst”
Request examples – Operational analysis
COM INTRO 2016: MARS Introduction and basic concepts 47
Slide 48
Retrieval of 2mt observation feedback from conventional data for 12 UTC analysis run on 1 February 2015.
retrieve,class = od,stream = oper,expver = 1,date = 20150201,time = 12,type = ofb,obsgroup = conv,filter = “select lat,lon,obsvalue where varno=39”,target = “2mt.odb”
Request examples – ODB observation feedback
COM INTRO 2016: MARS Introduction and basic concepts 48
Slide 49
directives from input stream
directives from file
Retrieving data – Calling MARS in a script mars <<EOFretrieve,type = an,date = -1, target = “$SCRATCH/my_an”
EOF
cat > my_request <<EOFretrieve,type = an,date = -1, target = “$SCRATCH/my_an”
EOFmars my_request
COM INTRO 2016: MARS Introduction and basic concepts 49
Slide 50
Point your browser to
or on software.ecmwf.int navigate to
User Documentation > MARS user documentation > MARS example requests
and follow the instructions
MARS Practical
software.ecmwf.int/wiki/display/UDOC/MARS+example+requests
COM INTRO 2016: MARS Introduction and basic concepts 50
Slide 51
Default values: minimize their use
No semantic check (only syntax is checked)
MARS messages- INFO request execution and report- WARNING unusual aspect of execution- ERROR system or data errors- FATAL terminates execution
Retrieving data – Hints
COM INTRO 2016: MARS Introduction and basic concepts 51
Slide 52
apps.ecmwf.int
COM INTRO 2016: MARS Introduction and basic concepts 52
Slide 53
Web interface to entire archive content
Content browsing of every field in the archive- more up to date than static content documentation
URL based on MARS requests (can be edited & bookmarked) Real-time (dynamic access to metadata) Create MARS requests (without checking availability) Check availability of data Retrieval in GRIB and NetCDF for few fields
Web-MARS – apps.ecmwf.int/mars-catalogue/
COM INTRO 2016: MARS Introduction and basic concepts 53
Slide 54
Server activity / MARS queue Show system activity Monitor your requests Learn how the queuing system works
- Reason for queued requests
Web-MARS – apps.ecmwf.int/mars-activity/
COM INTRO 2016: MARS Introduction and basic concepts 54
Slide 55
GRIB table based view Links to IFS documentation Links to comprehensive list of class, stream and type
https://software.ecmwf.int/wiki/display/GRIB/Documentation
Parameter database - apps.ecmwf.int/codes/grib/
COM INTRO 2016: MARS Introduction and basic concepts 55
Slide 56
Some useful tools
grib_ls, grib_dump, …
Metview examiners
- metview -e <grib|bufr|netcdf|odb> <file>
CDO - Climate Data Operators
- See https://code.zmaw.de/projects/cdo
Retrieving data – Helpers
COM INTRO 2016: MARS Introduction and basic concepts 56
Slide 57
GRIB API tool grib_to_netcdf
To convert a GRIB file to NetCDF format
GRIB must be a regular lat/lon grid or a regular Gaussian grid
- i.e. the key "typeOfGrid" should be "regular_ll" or "regular_gg"
Example> grib_to_netcdf -o output.nc input.grib1
Retrieving data – Conversion to NetCDF
See https://software.ecmwf.int/wiki/display/GRIB/grib_to_netcdf
COM INTRO 2016: MARS Introduction and basic concepts 57
Slide 58
Client/Server
Protocol: MARS request
Clients, C program + GRIB API + libemos library (Interpolation) - Supercomputers- Workstations and Servers- Applications like Metview (local / at ECMWF)- Remote client for Member States (security mechanism)- WebMARS- Data Server- Web API
MARS Architecture
COM INTRO 2016: MARS Introduction and basic concepts 58
Slide 59
Reports Database (RDB), on-line observations (for Operations only) Fields Database (FDB)
- Data produced by most recent cycles or experiments- Very fast access (on-line data)- Suitable for model input
ODB server, on-line ODB on supercomputers Main Archives (multiple servers)
- Dedicated Linux servers / clustered architecture - Terabytes of disk space- Tape management SW: HPSS- Oracle (Sun) SL8500 Automated Tape Libraries
MARS Architecture – Servers
COM INTRO 2016: MARS Introduction and basic concepts 59
Slide 60
1) Check syntax (MARS language and request syntax)
2) Print request to be processed
3) Query all Supercomputer’s FDB
4) Query main archives (if data not in FDB)
5) Transfer data
6) Post-processing while transferring (if needed)
7) Report on result
MARS Architecture - Request execution
COM INTRO 2016: MARS Introduction and basic concepts 60
Slide 61
Request execution (1/3)MARS - INFO - **MARS - INFO - **
PPDIR is /ppdir/data/rs60005mars - INFO - 20090225.102926 - Welcome to MARSretrieve,
class = od,type = an,expver = 1,date = -7,time = 00/to/18/by/6,param = t,levtype = model level,levelist = 1/to/91,area = E,grid = 2.5/2.5,target = "t.ll"
mars - INFO - 20090225.102942 - Processing request 1mars - WARN - 20090225.102942 - Area not compatible with gridmars - WARN - 20090225.102942 - Area changed from 73.5/-27/33/45 to 75/-27.5/32.5/45
COM INTRO 2016: MARS Introduction and basic concepts 61
Slide 62
Request execution (2/3)RETRIEVE,
CLASS = OD,TYPE = AN,STREAM = DA,EXPVER = 0001,REPRES = SH,LEVTYPE = ML,LEVELIST = 1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/
24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/91,
PARAM = 130,DATE = 20090218,TIME = 0000/0600/1200/1800,STEP = 00,DOMAIN = G,TARGET = "t.ll",RESOL = AUTO,AREA = 75/-27.5/32.5/45,GRID = 2.5/2.5,PROCESS = LOCAL
COM INTRO 2016: MARS Introduction and basic concepts 62
Slide 63
Request execution (3/3)
mars - INFO - 20090225.102942 - Requesting 364 fields819480 FDB; INFO; DB$_ Fields DataBase 4.2mars - INFO - 20090225.102942 - Calling mars on 'marsod', callback on 61767mars - INFO - 20090225.104347 - Mars client is on ecgate.ecmwf.int (136.156.240.111) 61767mars - INFO - 20090225.104347 - Mars server is on hdr16.ecmwf.int (136.156.228.176) 57793mars - INFO - 20090225.104347 - Server task is 526 [marsod]mars - INFO - 20090225.104347 - Request cost: 364 fields, 445.507 Mbytes online [marsod]mars - INFO - 20090225.104347 - Transferring 467148136 bytesmars - WARN - 20090225.104348 - INTFB: Resolution automatically set to 63mars - INFO - 20090225.104423 - 364 fields retrieved from 'marsod'mars - INFO - 20090225.104423 - 364 fields have been interpolated on 'ecgate'mars - INFO - 20090225.104423 - Request time: wall: 14 min 42 sec cpu: 12 secmars - INFO - 20090225.104423 - Read from network: 445.51 Mbyte(s) in 24 sec [18.43 Mbyte/sec]mars - INFO - 20090225.104423 - Processing in marsod: wall: 14 min 6 secmars - INFO - 20090225.104423 - Visiting marsod: wall: 14 min 42 secmars - INFO - 20090225.104423 - Post-processing: wall: 11 sec cpu: 9 secmars - INFO - 20090225.104423 - Memory used: 13.48 Mbyte(s)mars - INFO - 20090225.104423 - No errors reported
COM INTRO 2016: MARS Introduction and basic concepts 63
Slide 64
Request scheduling Queuing system
Priorities: user, request age, request cost (number of tapes and fields)
Data collocation MARS tree Archive objects (for OD data)
- 1 file per month of AN (1 level type, all times, levels, params)
- 1 file per forecast (1 level type, all steps, levels, params)
- 1 file per EPS (1 level type, all steps, members, levels, params)
- 1 file per month of ERA Interim FC (1 level type, all levels, times, steps, params)
Retrieving data
COM INTRO 2016: MARS Introduction and basic concepts 64
Slide 65
Retrieving data - MARS tree
COM INTRO 2016: MARS Introduction and basic concepts 65
Slide 66
Conversions
- SH → SH (reduced truncation), GG, LL
- GG (reduced) → GG (lower resolution or regular), LL
- LL → LL (lower resolution)
Sub-area extractions (GG, LL, waves), reduces data volume Derived fields (e.g. U and V from vorticity and divergence) Rotation
Retrieving data - Post-processing
COM INTRO 2016: MARS Introduction and basic concepts 66
Slide 67
Truncation before interpolation, reduces necessary resources
Retrieving data - Post-processing
Grid increment Truncation2.5 ≤ Δ T63
1.5 ≤ Δ < 2.5 T106
0.6 ≤ Δ < 1.5 T213
0.4 ≤ Δ < 0.6 T319
0.3 ≤ Δ < 0.4 T511
0.15 ≤ Δ < 0.3 T7990.09 ≤ Δ < 0.15 T12790.0 ≤ Δ < 0.09 T2047
COM INTRO 2016: MARS Introduction and basic concepts 67
Slide 68
Use local disk ($SCRATCH)
Estimate amount of data (list command)- Number of fields (up to tens of thousands / request)
- Data size (up to several Gigabytes / request)
Check computer resources: quota, CPU time, …
Reduce number of tapes involved (better scheduling)
Retrieve as much data from the same tape as possible
Avoid constantly accessing the same tape
Do not create unnecessary sub-archives
Retrieving data – Efficiency
COM INTRO 2016: MARS Introduction and basic concepts 68
Slide 69
Archived data- Available to all registered users
Current (valid) data, i.e. data for which the value of
(DATE + TIME + STEP) + 24 hours ≥ current date/time
- Needs special registration - Contact your Computing Representative
Boundary Conditions Project & COSMO-LEPS- Restricted to participating MS / individual users
Restrictions for Observations, TIGGE, EUROSIP… Data is available according to dissemination schedule, see
- www.ecmwf.int/en/forecasts/documentation-and-support/data-delivery/dissemination-schedule
For time-critical retrievals, use time-critical framework (option 1)
Retrieving data – Data access
COM INTRO 2016: MARS Introduction and basic concepts 69
Slide 70
Public distribution of data (licensing depends on datasets)
- Self-registration
Based on ecCharts framework
Data Server – http://apps.ecmwf.int/datasets/
COM INTRO 2016: MARS Introduction and basic concepts 70
Slide 71
To access MARS and ECMWF Public Datasets in batch- Delivers data directly to the users’ machine
To replace local MARS clients Alternative to retrieve–transfer jobs on ecgate Requirements
- Computer or Web User ID- client library, e.g. python - API key
Web API
See https://software.ecmwf.int/wiki/display/WEBAPI/Access+MARS
COM INTRO 2016: MARS Introduction and basic concepts 71
Slide 72
The Web API is a thin client
Does not use GRIB API nor any interpolation library- therefore any functionality requiring decoding of fields (e.g. compute,
read, write, multi-target) is not supported
Only one request per MARS call is recommended
Provides access to the Archives
Web API
COM INTRO 2016: MARS Introduction and basic concepts 72
Slide 73
Data Server – Web API
COM INTRO 2016: MARS Introduction and basic concepts 73
Who Data discovery Access method
Public user Public Datasets Access ECMWF Public Datasets
Commercial user Archive catalogue Access MARS
Authorized user MARS catalogue Access MARS
https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home
Slide 74
Web API Python example
COM INTRO 2016: MARS Introduction and basic concepts 74
#!/usr/bin/env pythonfrom ecmwfapi import ECMWFService
server = ECMWFService("mars")server.execute(
{ "class": "od", "date": "20160101", "expver": "1", "levtype": "sfc", "param": "167.128", "step": "0/to/240/by/12", "stream": "oper", "time": "00", "type": "fc" }, "target.grib")
Examples for retrieving large datasets efficiently:
https://software.ecmwf.int/wiki/display/WEBAPI/Retrieval+efficiency
Slide 75
MARS documentationsoftware.ecmwf.int/wiki/display/UDOC/MARS+user+documentation
Web-MARSapps.ecmwf.int/mars-catalogue/apps.ecmwf.int/mars-activity/
FAQ http://www.ecmwf.int/search/faqs
ECMWF real-time datasets www.ecmwf.int/en/forecasts/datasets
IFS Documentation www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model/ifs-documentation
GRIB API Documentation software.ecmwf.int/wiki/display/GRIB/Home
Additional resources
COM INTRO 2016: MARS Introduction and basic concepts 75
Recommended