Inspire Compliant Weather Data

INSPIRE Compliant Weather DataRoope Tervo, Mikko Visa, Jukka Pakarinen | Finnish Meteorological Institute

05/03/2023 INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka Pakarinen

2

Overview

Data Models

Producing Data Sets

Experiences


3

•Finnish Meteorological Institute opened its data in 2013.•Basically everything FMI owns was opened.•The very same data portal works as Open Data and INSPIRE portal.

FMI Open Data

Meta data ServicesISO19115 WFS WMS

CSW

Grid Series Observations

Time Series Observations

Data Models O&M

Simple Feature

GRIBNetCDF GeoTiff

05/03/2023 4INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka Pakarinen

Data Sets•Way beyond INSPIRE Data Specification scope

• Instantaneous weather and marine observations

• AWS, soundings, mast, air quality, sun radiation, marine, lightning…

• Radiation observations by Radiation and Nuclear Safety Authority (STUK)

• Road weather observations

• Model data from 6 different weather and marine models

• Weather radar images

• Climatological data

• Observation time series, analysis, climatological reference…


11

Overview

Data Models

Producing Data Sets

Experiences

05/03/2023 12

Data Modelso Observations and point

forecasts as GMLo The same data is published in

three different formats.o Gridded data is provided in

appropriate binary format (grib, NetCDF, HDF…)o WFS members contains the

metadata ‘envelope’ with a link to a actual data

o WCS will replace this ‘arrangement’ at some point

INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka Pakarinen

o Every XML document as self-explanatory as possible


13

om:phenomenonTime• Tells start and end time of returned data

Data Modelsmost important elements


14

om:procedure• Describes data producing procedureom:parameter• Changing parameters in the process

• i.e. analysis time of the numerical model



15

om:featureOfInterest• Describes location(s) or area of returned data• Several different kind of identifiers

• target:SurfaceWeatherTargetArea area (grid data)

• fmisid FMI id for observation station

• wmo WMO id for observation station

• geoid http://geonames.org id for location

• name Human readable name

• region county of the station


http://geonames.org/


16

om:observedProperty• Describes meteorological parameters in returned data• Defined as xlink• Describes label, basePhenomenon and units of measure of

every parameter in the reponse• Tip: use the same service without parameter definitions to

query all possible parameters• http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast

• Note that all parameters are not available in all data sets


http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast


17

om:result• Data in required format in observations and point forecasts and link to a

external binary data in grid formatted data


05/03/2023 18

Data Modelsgmlcov:MultiPointCoverage

gml:rangeSetgml:doubleOrNilReasonTupleList

The data is listed for every point defined in domain set.

gml:domainSetgmlcov:simpleMultiPoint

The coverage is defined as a list of

points in 4 dimensional grid (lat,

lon, height, time).

gmlcov:rangeType

The parameters listed in range set

are defined in separate element.


05/03/2023 19

Cons- Not intuitive- No natural

structure of XML XSLT and Xpath don’t work

Pros+ Compact+ Efficient+ Small file size+ Works for many

data types

Data Modelsgmlcov:MultiPointCoverage


05/03/2023 20

Data Modelswml2:MeasurementTimeseries

wml2:MeasurementTimeseries

One member contains time series of one parameter and

one location


05/03/2023 21

Cons- Lots of repetition- Large file size- Heavy for DOM-

based parsers- Don’t work i.e. for

thunder strikes

Pros+ Intuitive+ Easy to use+ XSLT & XPath

works

Data Modelswml2:MeasurementTimeseries


05/03/2023 22

Data ModelsSimpleFeature

SimpleFeature

One member contains one time step of one parameter

and one location


05/03/2023 23

Cons- Huge file size- Heavy for DOM-

based parsers

Pros+ Intuitive+ Easy to use+ Wide application

support

Data ModelsSimpleFeature


05/03/2023 24

Data Models file size comparison

Simple

Featur

e

Measu

remen

tTim

eseri

es

MultiPoin

tCov

erage

0

20

40

60

80

10081.7

52.9

1.81.3 1.2 0.2

Document Size [MB]

Compressed DocumentSize [MB]

o 138 o

bser

vatio

n stat

ions

o 11 pa

ramete

rs

o 72 tim

e step

s (12

hour

s)


05/03/2023 25

Data Models Popularity Comparison

0

20

40

60

80

10080

19.8

0.2

Downloads[%]

o Based

on on

e mon

th da

ta (0

4/201

4)

o All data

sets

combin

ed


Data Type Data Format

Observations wml2:MeasurementTimeseries

gmlcov:MultiPointCoverage

SimpleFeature

Point Forecasts wml2:MeasurementTimeseries

gmlcov:MultiPointCoverage

SimpleFeature

Lightning Observations gmlcov:MultiPointCoverage

SimpleFeature

Grid Forecasts XML Envelope + Grib2/NetCDF

Radar Images GeoTiff / PNG images

METAR IWXXM



27

• Aviation weather reposts are delivered as IWXXM• New data model coming into use in aviation defined by

ICAO and WMO

• Consists of the same elements than other messages• om:phenomenonTime, om:procedure, om:featureOfInterest,

om:result

• Content of the METAR is in om:result part as• extracted into XML elements

• original, “old fashion”, METAR string

Data Modelsaviation observations IWXXM / METARS

http://en.wikipedia.org/wiki/IWXXM


28

• Once published, content and structure of IWXXM messages will not change

• Messages will contain a digital signature

• Use GetPropertyValue to download only original METAR string

• http://data.fmi.fi/fmi-apikey/…/wfs?request=GetPropertyValue&storedquery_id=fmi::avi::observations::finland::latest::iwxxm&valuereference=wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/avi:metadata/avi:MessageMetadata/avi:source/avi:Process/avi:input


http://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetPropertyValue&storedquery_id=fmi::avi::observations::finland::latest::iwxxm&valuereference=wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/avi:metadata/avi:MessageMetadata/avi:source/avi:Process/avi:input





















29

• Messages have been automatically created based on the original METAR/SPECI code messages.

• Messages are NOT an authoritative aviation weather report. They SHOULD NOT be used as a weather report to be used in flight planning or other aviation related use.



30

• WFS response contains the same meta data information than in observations and point forecasts

• om:result contains gmlcov:RectifiedGridCoverage• Basically the same with multipointcoverage

• domainSet defines the grid (now as regular grid)

• rangeSet contains data as an external link to the binary content• rangeType defines the parameters

Data Modelsbinary data


31

• Note that fileReference points often to an other service• Can be used directly, but

• It is always good practice to consult WFS for available times and parameters


Client

WFS

WMSFMI Data Server

Available radar images?Available weather model outputs?


32

• Area, time and weather parameters may be defined in the request

• For weather models, intersection of requested area and available data area is returned

• For radar images, all images which intersects requested spatial and temporal space are returned


Requested area

Radar images

Radar images Requested area

Model coverage

Weather models


33

Radar images• The reference points to a “original” gray scale GeoTiff image• Images can also be downloaded as color images

• Remove &styles=raster from the data request

• But then information is lost

• Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld

• Consult om:parameter element for single radar measurement parameters

• Scanning angle, bin count and bin length


https://github.com/fmidev/opendata-resources/tree/master/sld

https://github.com/fmidev/opendata-resources/tree/master/sld


34

Numerical models• The reference points to (often a subset) the model output in

appropriate format• GRIB2 for weather models

• NetCDF for marine models

• Note that whole model output can be large• Up to 17 GB

• Download always only area, parameters and time range of interest


05/03/2023 35

ef:EnvironmentalMonitoringFacilityData ModelsEnvironmental Facilities(blueprint)

ef:EnvironmentalMonitoringNetwork

(AWS | Radar | ….)

• Activity time• Procedure• Observed property (same with

data products)


ef:observingCapability 1…*

ef:belongsTo(0…*)

(link to W

FS response)

contains (0…*)

(link to WFS response)


36

Overview

Data Models

Producing Data Sets

Experiences


37

Services

Data (NFS) Databases

GeoNetwork

CSW

GeoServer

WMS

SmartMet Server

WFS & WCS & WMS•

Most c

ommon

layers

•W

FS Bas

ed on

stored

queri

es

•Sem

i-auto

matic

sync

ing fro

m datab

ase

Open Data Service Cluster

S1 S2 S3

Client Data Service Cluster

S1 S2 S3

Load Balancer

Configuration

Data (NFS)

Configuration (NFS)Database

Clustering


05/03/2023 39

Weather Data – Volumes• In-situ weather measurement 1 TB• Weather radar data 50 TB• NWP model gridded data (FMI)

• HIRLAM 120 TB• AROME 230 TB

• Satellite image data (FMI)• Globsnow 32 TB• Other 50 TB

• Climate model data (FMI) 29 TB• Other models

• SILAM 100 TB• Tuuliatlas 21 TB• Other 300 TB


05/03/2023 40

Producing INSPIRE Data ProductsObservations


Database

Master database

SmartMet Server

In-memory database

latest data. Server provides

logic for fetching the data from

local or master database.

SmartMet Server WFS

On-the-fly formatting

(based on template)

INSPIRE compliant

XML response

05/03/2023 41

Producing INSPIRE Data ProductsPoint Forecasts


File System

Forecast model data

SmartMet Server

Memory mapped data.

Server provides logic for

interpolation of data for

requested area and time.

SmartMet Server WFS


(based on template)

INSPIRE compliant

XML response

Producing INSPIRE Data ProductsGrid Forecasts 1/2

File System

Forecast model data

SmartMet Server

Server provides relevant meta

data to fetch the data content

SmartMet Server WFS


(based on templates)

INSPIRE compliant

XML response

containing meta data

and link to binary data


Producing INSPIRE Data ProductsGrid Forecasts 2/2

File System

Forecast model data

SmartMet Server

SmartMet Server Download

Format the data to

requested format and

projection.

Binary data

Memory mapped data.

Server provides logic for

interpolation of data for

requested area and time.



44

SmartMet Server•Data and product server for MetOcean data•High capacity & availability

• FMI installation handles over 30 000 000 requests each day

•Data is extracted and products generated on-demand•INSPIRE Compliant•Operative since 2008

• FMI client services (since 2008)• Finnish Meteorological Institute (FMI) Open Data Portal (since 2013)• Going to be used at Copernicus C3S Climate Data Store (ECMWF)

•Open source

C++


45

SmartMet Server•Several input sources

• GRIB-, NetCDF-, etc. files (multi-dimensional grid data)

• PostGIS database (vectors)

• Point database (point observations)

•Several output interfaces and formats• WMS, WFS 2.0

• JSON, XML, ASCII, HTML, SERIAL

• GRIB1, GRIB2, NetCDF

C++


46

SmartMet ServerUsage

FMI Open Data Portal & FMI INSPIRE Data Services

Backend for clients’ web services

Integration to clients’ systemsBackend for

mobile applications

Backend for FMI Client Services

Backend for FMI public pages

• Basis of most FMI product generation

• Published in 2016 in GitHub• https://github.com/fmidev/smartmet-server

• MIT Licence• Documentation in GitHub

• FMI will host the development• Small contributions with pull requests

• In larger contributions, implementation plan is recommended (in GitHub wiki)

• CLA (Contributor Licence Agreement) will be required

05/03/2023 INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka Pakarinen 47

SmartMet ServerOpen Source

https://github.com/fmidev/smartmet-server



Interested?https://github.com/fmidev/smartmet-server

Source code and documentation

Follow: https://github.com/fmidev

https://facebook.com/fmibeta

New releases will be updated here

05/03/2023 INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka Pakarinen 48


https://github.com/fmidev



53

Overview

Data Models

Producing Data Sets

Experiences

05/03/2023 54

INSPIRE Data SetsHow to define a data set?o All weather observations from

Finland? Would cause over 50 000 000 Observations (XML file size ~37 G)

o All observations from one observation station? Would cause over 200 data sets

o Even one year’s observations cause too large data set to handle


05/03/2023 55

INSPIRE Data SetsMeteorological data is a constant flow of observationsFMI has one data set per data type, i.e. one for ground weather, observations, one for Hirlam weather forecasts, etc…Every data set have predefined area and time range.


05/03/2023 56

INSPIRE Data SetsIt is notable that data set response depends on time it’s requested Unique identifiers are not

reasonable


05/03/2023 57

And a little over450 000 data downloads

per day (5,2 req/s)

At the moment over 11 000

registered users

Some Experiences


05/03/2023 58

Practically no client supports complex features

Although standards are followed, there’s

a gap between provided data model

and clients’ capabilities


Some Experiences

05/03/2023 59

GeoServer was modified to support stored queries in

WFS 2.0 (released in version 2.7)

FMI published the same data as simple features to support

clients


Some Experiences

05/03/2023 60

Industry is happy to use standardized

services

Amateur and freelancer coders

would prefer simple JSON API


Some Experiences

05/03/2023 61

…but suites quite well for exchanging (subsets of) data.


Data format is too verbose for clients to

use directly…

Some Experiences

05/03/2023 62

For now, very few have

been interested in forecast models as a grid data

Point forecasts, observations and

radar images are the most interesting data

types


Some Experiences

www.fmi.fi

http://ilmatieteenlaitos.fi/avoin-lahdekoodihttps://github.com/fmidev

https://en.ilmatieteenlaitos.fi/open-datahttps://facebook.com/fmibeta

http://roopetervo.comhttp://www.slideshare.net/tervo



https://en.ilmatieteenlaitos.fi/open-data

https://en.ilmatieteenlaitos.fi/open-data


http://roopetervo.com/

http://www.slideshare.net/tervo/



Engineering

Inspire Compliant Weather Data