1

T.C.TURKISH STATE METEOROLOGİCAL SERVICE

DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES

METEOROLOGICAL DATA MANAGEMENT

Mustafa Sert October 2011

2

OUTLINE

Meteorological data definition Observation Stage Telecommunication Stage Archive and Quality Control Stage Presentation Stage

3

Meteorological Data

Google definition;

“Meteorological facts pertaining to the atmosphere, such as wind, temperature, air density, and other phenomena that affect.”

Another definition;

“All observed, derived, calculated, visualized data for meteorological aims.“

Meteorological data is of vital importance used in scientific and social areas for a sustainable world. Meteorological data is the very basic input in diagnosing the climate change, global warming, weather forecasting, weather pollution, general atmospheric cycle, decision-make, etc..

4

Stages

Meteorological data production (Observation,Calculation, etc.) Telecommunication of Met.data Archive and quality control of Met. data Presentation of Met. data

5

Stage1; Meteorological data production

Meteorological data sources are; Observations of surface stations (Automatic Weather Observation Systems,

Manuel Observation Systems, etc.) Observations of upper-Air Stations (Ravinsonde, etc.) Observations of sea Stations (Coastal Stations and ships, buoys) Meteorological Radars Meteorological Satallites Meteorological Models (Forecast model, Climate model, Dust model, etc.)

6

Stage1; Meteorological data production

TSMS Observing System;

132 Synoptic Stations 257 Climatological Stations (242 AWOS, 15 Manuel) 63 Aviation Stations (50 AWOS, 13 Manual) Total 463 AWOS Stations, 246 installing (2012) 40 Sea Stations (installing, 2012) 8 Ravinsonde Stations 8 Meteorological Radars ( 4 installing)

7

Stage1; Meteorological data production

TSMS AWOS Observation Network

8

Stage1; Meteorological data production

For good-quality Observations;

Select the correct location. Use WMO Standards. Select the correct instruments. Use WMO Standards. Specify the data production standards. (For AWOS, do you need the minutely

data for pressure or temperature?) Specify the station limits for quaility control process. Create metadata database and open it to users.

9

Stage2; Meteorological data Communication

It is important to transport meteorological data from station to centre correctly and in time.

Specify which communication systems you use. It is related to Country’s telecommunication infrastructure.

Specify alternative communication systems. At least 2 communication system is offered for backup.

10

Stage2; Meteorological data Communication

TSMS Communication Network includes;

110 VSAT Connection 5 Central Satallite Receiving System (2 EumetCast, 1 L-X Band, 2 L Band) and 4

Satallite Receiving System (EumetCast) 340 GPRS Connection 330 ADSL Connection 8 radio-link connection 150 Mb MetroEthernet Internet connection (Ankara - Centre) 10 Mb MetroEthernet Internet connection (İstanbul - Region) PSTN (Dial-up) connection Fax connection

11

Stage2; Meteorological data Communication

12

Stage3; Archive and Quality Control

Our data archiving strategy is ;

For small data like observations ; To use relational databases with fast disk systems.

For huge data like radar, satallite and model ; To use storage managemant software with slow & big capacity disk - tape systems.

13

Stage3; Archive and Quality Control

Other basics of data archiving;

• Collect all data in one center and one disaster recovery center for easy management

• Virtualization technology for database servers may be a good solution.

• Enterprise tape library for huge data is optimal solution.

14

Stage3; Archive and Quality Control

What about the model data ? (like climate models)

• Climate model outputs consist millions of files. Total volume reaches to terabytes.• It is very important to access these outputs fast and reliable.

Solution ?

Data Aging policy !! ( Bring frequently-used, new data to the fast storage and move old, least-used data to the slow – big capacity storage)

Use fast disks for frequently-used climate model outputs. Use slow&big capacity disks and enterprise tape librarys with storage

management software.

15

Stage3; Archive and Quality Control

What we do ?

Our Harware and Software platform for climate model is ;

HP DL580G5 Server (4 x 2.4 Ghz Intel Xeon Processorwith dual core, 8 GB RAM, 420 GB Internal FC Disk, 12 TB External SATA disk)

SLES 10 ES Operating System

After a period of time;

We archive old data in IBM TS3500 Tape Library with

IBM Tivoli Storage Manager Software.

TS3500 Tape Library has ; 12 x IBM 3592 700 GB Teyp Drive,

Total 290 TB Tape Media (415 Data Cartridge x 700 GB)

16

Stage3; Archive and Quality Control

Tivoli Storage Manager Software (TSM) provides archiving all kind of data from Model server’s command prompt.

To archive model output from command line use ;#dsmc arc “/home/precis/precis_out/iklao000.iklao.d10182.t110627.leave”

To retrieve model output from command line use ;#dsmc ret “/home/precis/precis_out/klao000.iklao.d10182.t110627.leave”

Tape Library Robotic System

17

Stage3; Archive and Quality Control

Data Quality Control;

If your input data is bad, your model outputs are bad as well!Observations should pass the data quality control alghoritms for healthy model inputs.

In quality control, we use; • Range Test, • Step Test, • Persistence Test, • Like Instrument Test and • Spatial Tests

18

Stage 4; Presentation

Providing data as fast as possible from archiving system to all internal and external users.

To do this, we use Web Portal. (tumas.dmi.gov.tr)

Web portal can access fast Sybase IQ datawarehouse (OLAP based Database) and MARS System for retrieve requested data.

Data output formats for observations from database may be xml, doc, xls,txt,pdf. Data output formats for observations from MARS may be bufr.

19

Questions?

msert@dmi.gov.tr

Turkish State Met. ServiceDepartment of Research And Information Technologies

Information Technology and Statistics Division October – 2011

Alanya