AMRC Meteorological Data Distribution System Part I:Data Sources and Archive

K. Elena Willmot, Matthew A. Lazzara, Linda M. Keller, Nicole M. Schroeder

Antarctic Meteorological Research CenterSpace Science and Engineering Center

University of Wisconsin-Madison

Introduction

o AMRC since 1992...o Part I Outline:

o Data SourcesInternal & External

o Data Formatso Data Archive

Historical & Present

o Distributiono Challenges

No more FTP?

Access and quality of data

Lack of acknowledgements

o Statistics

(Photo by Michael Forster Rothbart, UW-Madison University Communications)

Data Sources: Internal

o AWS (Automatic Weather Station) datao Significant portion of internal data comes from UW’s AWS networko Real-time and archive quality data sets

o Ten minute intervalso Hourly intervalso Three-hourly intervals

o 60+ weather stations across the continent

Data Sources: Internalo Antarctic Composites

o Conceived by UW’s Professor Charles R. Stearns in October 1992.o Cover South Pole region to about 40° South.o Satellites used:GOES, GMS, MTSAT, METEOSAT, FY-2, KALPANA,

NOAA, DMSP, Aqua & Terra, and MetOp

o Created every 3 hours/1 hour every dayo Currently, 4 sizes:

o 512 x 512o 683 x 683o 1024 x 1024 (standard)o 2048 x 2048

o 6 types:o Infraredo Water Vaporo Experimental Visibleo Long-wave (Coming soon!)o Short-wave (Coming soon!)o Pseudo Color

Making Antarctic Composite Satellite

Imagery:

Data Sources: Internalo University of Wisconsin’s Space Science and Engineering Center

(SSEC) Data Centero Surface synoptic observationso METARs (Meteorological Aviation Reports)o RAOB (Radiosonde Observations)o Ship/buoy observations

o Numerical Models from GFS

(Global Forecast System) and ECMWF

(European Centre for Medium-Range

Weather Forecasts). o Polar-orbiting LAC (Local Area Coverage) observations from NOAA

satellites

Data Sources: Externalo Participation from various researchers, USAP (United States

Antarctic Program) participants, and international groups within the Antarctic meteorological community

o International AWS (real-time only)o USAP research vessel observationso Field experiment radiosonde observationso Surface observations at:

o Palmer Stationo McMurdo Stationo South Pole Stationo Various other field camps

Lawrence M. Gould. Photo by: Al Hickeyhttp://photolibrary.usap.gov/

Data Flow

International AWS

USAP Research Vessels

McMurdo, Palmer & South

Pole Stations

WisconsinAWS

AMRC Office

SSEC Data Center

NOAAHRPT

DMSPRTD

Aqua/Terra DB (McMurdo only)

NOAAGAC/LAC

MTSAT

Meteosat (2)

GOES (2)

FY-2C

Kalpana-1

Aqua/Terra NASA/NOAA

Products:

Data Formats

Archive:o McIDAS

o AREAo MDo GRID

o ASCII Texto GIF, JPEG

Distribution:o McIDAS

o AREAo MDo GRID

o ASCII Texto netCDF o GIF, JPEG

Data Archive: Historical

o Since 1992...o Media used to archive data:

1. Magneto-optic disks2. Compact disks (CD)3. Digital video disks (DVD)4. 8 millimeter Exabyte tapes5. DDS3 tapes (4 mm tapes)6. 3590 IBM tapes7. LTO tapes

1

2

3

4

5

6

7

Data Archive: Presento RAID/5 disk arrays as well as DVDs are most prominent todayo FTP site most widely used

o Updated monthly witho AWS – SPAWAR obso USAP research vessel observationso McMurdo Station obso Palmer Station obso South Pole Station obs

o Summary / notification email sent out

Distributiono Data Services to be discussed in Part II:

o FTP (File Transmission Protocol)o Rsync Serviceo Antarctic-IDD (Internet Data Distribution) LDM (Local Data Manager)o ADDE (Abstract Data Distribution Environment)o E-mailo Websiteo Future:

o RAMADDA (Repository for Archiving, Managing and Accessing Diverse Data)o GIS (Geographic Information Systems) Software

Challenges:1o USAP moving away from FTP site – advancing security ruleso Don’t fret! FTP will still be available here at the AMRC:

o Other options: Link with external systemo Limited resources to maintain linkso Manually providing data is expensive and not desired

o We aim to provide data in a more semi-automated, self-service fashion.

Challenges:2o Accessing data externally

o Poor qualityo Corrupt fileso Missing

o Gaps in the archiveo Which data can be used / distributed and which cannot?

o Third party redistribution limitations

o Evolving toward nearly fully automated data flow system (from source to archive)o Maintenance will be a constant issue – especially at the sourceo Documentation

Challenges:3 - Metadatao Ever increasing need for Metadatao Significant effort to research/document and provideo Incomplete historical records, leave some metadata information

missing…o E.g. South Pole...

We have found an issue with the South Pole data:

From 1996 and earlier, the South Pole LCDs (Local Climate Data) used data that switched between using local time and GMT time when recording observation times.

South Pole is working on re-running past LCDs and we do not know how long it will take / when they will be completed.

Acknowledgements

The authors appreciate the support of the Automatic Weather Station Program and/or Antarctic Meteorological Research Center for the data set, data display, information, etc. ([AMRC/AWS staff member, as appropriate], NSF grants number ANT-0636873, and/or ANT-0838834)

Sculptor Gabriel Warren used a transparency of the B-15A Iceberg in one of his pieces called “Interrogation: Dialogue” in his “Rhegmalogia” collection.

http://www.art-farm.net/homeGabe.htm

AMRC/AWS Statistics

o Show Gigabytes transferred / month on Website.o Average of 70-80 Gb year-round.

o Gigabytes transferred / month on FTP site.o Generally very variable – maybe due to field season.

• ANTIDD – Antarctic-IDD mailing list• ANTDISCUSSION – Antarctic meteorology

discussion group• ANTAWS - Information and updates on

AWS data• ANTOBS - Information and updates on all

other observational datasets• ANTOPS - Notices on AMRC’s real-time

data availability, distribution and status• ANTICEBERG - Notices on latest Iceberg

monitoring images posted to the web site• ANTCOMMUNITY - Infrequent notices to

the Antarctic community on events and significant developments

Please join our mailing lists!:

How to subscribe:Visit our new website and click on the “Subscribe” button

next to the mailing list you wish to join:

THANK YOU!

Acknowledgements:

National Science Foundation Office of Polar Programs Grant #0838834South Pole Meteorology Office/Raytheon Polar ServicesSPAWAR Office of Polar ProgramsMcMurdo Station Research AssociatePalmer Station Research AssociateUSAP Research Vessels – LM Gould & NB PalmerMcMurdo Ground StationInternational partners& More

Questions?