Data centre support for the IGS-RT PP

W. Söhne, H. Habrich, G. WeberFederal Agency for Cartography and Geodesy,

Frankfurt am Main, Germany

Outline

Introduction IGS-RT PP

• Data and product file centre• File types under consideration

Technical aspects• Data volume• Up- and downloading• Completeness • Limitations

Policy aspects• Upload policy

Conclusions & open questions

Introduction

Storing of highrate files nothing new for IGS• At CDDIS back to 2001, doy121• Initiative from the LEO needs• At IGN• Others

Term “highrate” is a relative one• Usually means 1 Hz sampling rate• But higher sampling rates in use depending on the

application Currently ~110 stations at CDDIS Clear statement from IGS concerning the

needs of long-term archive still missing

IGS-RT PP (1)

One activity of the IGS-RT PP: Data and product file centre

File types under consideration• Navigational data (broadcast ephemeris)• Observational data

- highrate RINEX files- hourly and daily RINEX files

• Derived files (products)- Orbits or orbit corrections- Clocks or clock corrections- Troposphere & ionosphere parameters

IGS-RT PP (2)

Why generation of highrate files?• Currently, considerable number of users are

“feeding” near real-time applications • For future scientific analyses

What is “near real-time”?• Typically hourly applications, with 20-30 minutes

computation time two times per hour computation possible

Why 15 minutes files?• Reduction of file size compared to an hourly

highrate file• “Convention”

IGS-RT PP (3)

Is that interval small enough? Are there other intervals of interest, e.g. 5 minutes files?• Advantages

- Better distribution of uploading over time- Other intervals can easily be created/derived- File naming convention fits

• Disadvantages- Interval size not known in IGS- Number of files growing

Technical aspects: data volume

Data volume of highrate observational files• Example BUTE115A45.08O: 1 Hz 15 minutes

GPS+GLONASS RINEX file, # obs. types 8, 20 SV in view: 2.4 Mb

• Hatanaka+compress: 170 Kb 5.7 Gb per station and year for the compressed

files >> 500 Gb per year• Selection of IGS stations for high-rate storing?

Data volume of product files• Clocks: 1 Hz, 1 hour: 2.5 Mb, compressed 170 Kb

5.7 Gb per solution and year• …

Technical aspects: up- and downloading

Data upload of observational files• Example 250 stations, 24 files per day (hourly):

6000 files uploaded * 100 Kb 0.6 Gb per day• Example 110 stations, 96 files per day (highrate):

10560 files uploaded * 170 Kb 1.7 Gb per day Critical aspects are

• Is the data centre able to handle all incoming files within, e.g., 5 minutes after the full hour?If not spreading of upload over a certain time span?!

• Is the data centre able to handle the parallel download requests in peak periods?

Technical aspects: completeness (1)

“Completeness” covers the aspects• Number of observation types• Number of epochs• Number of SVs per epoch• Ratio observed / predicted number of observations

Completeness can be affected by• Outages of single stations data streams• Outages of the broadcasting system• Handling of unhealthy SVs• Limitation of supported observation types

Technical aspects: limitations (1)

“Limitations” cover the following aspects• Resolution of

- Phase- Code

• Support of - GPS L2C, L5- GLONASS- Galileo- SBAS

Technical aspects: completeness (2)

Different levels of comparison of original and accumulated files possible• Character by character

- Difficult due to roll-over phase values • Completeness

- Epochs missing?- SVs missing?

• Parallel analysis- Differences between the results

Technical aspects: completeness (3)

Hourly file BUTE115A.08O (TEQC) Highrate file BUTE115A00.08O (BNC)

Technical aspects: limitations (2)

ALME: file from TEQC vs. file from BNC (RTCM 2.3) after RNXSMT (BSW5.0)

Technical aspects: limitations (3)

WTZR: file from TEQC vs. file from BNC (RTCM 3.0) after RNXSMT (BSW5.0)

Technical aspects: completeness (4)

579

80110

58 5662

99,9 -100,0%

99,5 - 99,6%

50,0 - 90,0%

100,0%

99,8 - 99,9%

90,0 - 99,0%

< 50,0%

99,6 - 99,7%

99,0 - 99,5%

99,7 - 99,8%

168

50010

2135 21 99,9 -100,0%

99,5 - 99,6%

50,0 - 90,0%

100,0%

99,8 - 99,9%

90,0 - 99,0%

< 50,0%

99,6 - 99,7%

99,0 - 99,5%

99,7 - 99,8%

0 1

001 0

4

14

2 2 99,9 -100,0%

99,5 - 99,6%

50,0 - 90,0%

100,0%

99,8 - 99,9%

90,0 - 99,0%

< 50,0%

99,6 - 99,7%

99,0 - 99,5%

99,7 - 99,8%

Completeness of 24 hourly high-rateRINEX files from RTIGS streamscoming in via udpRelay

Completeness of 233 hourly high-rateRINEX files from RTCMv2 streamscoming in via NTRIP/TCP

Completeness of 711 hourly high-rateRINEX files from RTCMv3 streamscoming in via NTRIP/TCP

Technical aspects: completeness (5)

Missing SVs G06, G21, G26, G30

Streaming interrupted Longer sections w/o GLONASS

Policy aspects (1)

Station A

Station B

Station C

Station N

Data centre 1

Data centre 2

Data centre 3

User a

User b

User c

User m

Mirroring

Mirroring

„Traditional“ situation transferring hourly files, daily files, ephemeris files

file creation at the station

Policy aspects (2)

Possible situation using real-time data streams for RINEX file creation

Station A

Station B

Station C

Station N

Broadcaster 1 User a

User b

Data centre 1

transferring files transferring streams

or

file creation possible at the data centre, at the users site, at a third party

Conclusions & open questions (1)

Derivation of highrate RINEX files from real-time streams suitable tool

Current interval for highrate files 15 minutes – is that small enough?

Recommendation #1: Storing of 1 Hz 15 minutes files, at least for the long-term archive

Derivation of daily files from streams instead of ftp transfer, at least for new or proposed stations?

RT-IGS PP CfP found five candidates (BKG, CDDIS, GA, KASI, Univ. of Padova)

Conclusions & open questions (2)

Limitations acceptable? If yes, how to point the users to these limitations effectively?

Derivation of RINEX files from real-time streams possible for everyone – needs some regulation or clarification: Who is allowed to upload files derived from real-time data streams?

Recommendation #2: highrate file creation and upload to the GDC in one hand, ideally at the broadcaster’s side