SBAS and GBAS Integrity for Non- Aviation Users: Moving Away from "Specific Risk" ION ITM 2011San Diego, CA. 25 January 2011 Sam Pullen, Todd Walter, and

SBAS and GBAS Integrity for Non-Aviation Users: Moving Away from

"Specific Risk"

ION ITM 2011 San Diego, CA.

25 January 2011

Sam Pullen, Todd Walter, and Per Enge

Stanford University

[email protected]

25 January 2011 Integrity for Non-Aviation Users: Moving Away from "Specific Risk" 2

Motivation (1): SBAS and GBAS for Non-Aviation Users

• Where augmentation signals can be received, SBAS and GBAS benefits are available to all users.

• However, integrity algorithms in airborne MOPS are designed to support specific aviation applications.

– Resulting integrity protection levels are not well-suited for other classes of users

• Correcting this would increase the attractiveness of SBAS and GBAS to non-aviation transport users (auto, rail, marine) and others.


Motivation (2): Accuracy and Integrity

• Accuracy bounds (e.g., 95% vertical position error, or VPE) can be measured and modeled with high precision

• Integrity bounds (e.g., 10-7 vertical protection level, or VPL) cannot be– Lack of sufficient measurements

– Flaws in Gaussian extrapolations to low probabilities

– Dependence on details of failure models and assumptions

– Too little is known; too much is uncertain…

Illustrative example – not

to scale or direction

95% HPE

HPL (per MOPS)

HPL (non-aviation

application)


WAAS VPE vs. VPL from FAA PAN Data (3rd Qtr 2010: July – Sept.)

Source: WAAS PAN Report #34, Oct. 2010.

http://www.nstb.tc.faa.gov/ DisplayArchive.htm

VP

L (

m)

VPE (m)

Max. VPE 7 m (at Barrow,

AK)

95% VPE 1.2 m

99% VPE 1.6 m

http://www.nstb.tc.faa.gov/DisplayArchive.htm





WAAS Reference Station Classifications (for this study only)

Figure source: FAA GNSS Press Kithttp://preview.tinyurl.com/4ofdzz4

7 Inner Stations

13 Outer Stations

18 Remote Stations


Max. VPE and VPL from FAA PAN Data (1 Jan. 2004 – 30 Sept. 2010)

10 15 20 25 30 350

5

10

15

20

25

30

35

40

45

50

VP

E o

r V

PL

(m

eter

s)

Quarterly PAN Report Number (8 – 34)

Max. VPL

Max. VPE

95% VPE

Worst Case Between “Inner” and “Outer” WAAS Stations “InOut” Set


Max. HPE and HPL from FAA PAN Data (1 Jan. 2004 – 30 Sept. 2010)

Worst Case Between “Inner” and “Outer” WAAS Stations “InOut” Set

HP

E o

r H

PL

(m

eter

s)

Quarterly PAN Report Number (8 – 34)10 15 20 25 30 35

0

5

10

15

20

25

30

35

40

45

Max. HPL

Max. HPE

95% HPE

One unusual result: 12 m

error at Cleveland in Spring 2005

(correct number?)

As expected, both HPE and HPL are

significantly lower than VPE and VPL.


Ratio of Max. VPL and Max. VPE from FAA PAN Data (“InOut” Station Set)

Quarterly PAN Report Number (8 – 34)10 15 20 25 30 35

0

1

2

3

4

5

6

7

8

9V

PL

/ V

PE

Rat

io f

or

Max

. Cas

es

Mean Ratio 5.38

Noticeable improving trend likely due to error reduction at individual

WAAS reference stations.

Less error reduction after PAN #20 (March 2007).


Ratio of Max. HPL and Max. HPE from FAA PAN Data (“InOut” Station Set)

Unusual error at Cleveland (if correct) just

barely exceeded by HPL.

HP

L /

HP

E R

atio

fo

r M

ax. C

ases

10 15 20 25 30 350

1

2

3

4

5

6

7

8

9

10

Mean Ratio 5.21

Weaker but visible improving trend – more variability.



How Many Samples Were Collected?

4.25 10-9 sec

(49,324.6 days)

(105.04 years)

All validated PAN data from 1 Jan. 2004 to 30 Sept. 2010

Assume data correlated

over 30 sec

1.4 10-8 independent

samples

Assume data correlated over 150 sec (~ one

CAT I approach)


samples

Assume data correlated over 600 sec (10 min)


samples


Average vs. Specific Risk Assessment

• Average Risk (my definition): the probability of unsafe conditions based upon the convolved (“averaged”) estimated probabilities of all unknown events.

– Probabilistic Risk Analysis (PRA) is based on this procedure

– Risk aversion and value of information (VOI) are applied to the outputs of PRA integrity risk requirements, alert limits

• Specific Risk (my definition): the probability of unsafe conditions subject to the assumption that all (negative but credible) unknown events that could be known occur with a probability of one.

– Evolved from pre-existing FAA and ICAO safety standards

– Risk aversion and VOI and buried inside specific risk analysis

– Results (risk and protection levels) are inconsistent with PRA


Simplified Example: Ionospheric Spatial Decorrelation (1)

20:15 UT 21:00 UT

Severe Ionospheric Storm Observed over CONUS on 20 November 2003



• Using PRA, estimated “prior” probabilities of severe decorrelation are combined with the likelihood of SBAS or GBAS mitigation to derive resulting user risk. – Prior probabilities need not be known precisely

– Benefits of improved mitigation (“better information”) appear naturally as lower integrity risk.

• Under FAA interpretation of Specific Risk, worst-case iono. delay gradient is “credible” and thus is assigned a probability of one.– Worst-case for GBAS (CAT I): an extremely large gradient that

escapes detection by “matching speed” with ground station » This differs in real time for each site and GNSS geometry

– Worst-case for SBAS (LPV): a very large gradient that is just small enough to avoid detection by master station



0 5 10 15 20 25 30 35 40 450

0.02

0.04

0.06

0.08

0.1

0.12

0.14

User Vertical Position Error (meters)

PD

F

Worst-case error, or

“MIEV”, is 41 m

Most errors are exactly zero due to ground detection and exclusion, but all zero errors

have been removed from the histogram.

Simulated results for Memphis GBAS impacted by severe ionospheric gradient(RTCA 24-SV GPS, 6-km, User-to-ground separation, 1 and 2-SV impacts)

Most plotted (non-zero) errors are below 10 m even under severe conditions.


Benefits of an “Average Risk” Approach (Potential SBAS PL Reduction)

• “Average risk” approach supports large reductions in HPL and VPL implied by WAAS PAN data, pending more complete database analysis.

• Use “full-scale” PRA to re-assess “rare-normal” and faulted errors.

24 25 26 27 28 29 30 31 32 33 340

5

10

15

20

25

30

35

40

PAN Report Number

VP

L o

r H

PL

(m

ete

rs)

95% VPL

95% HPL

Adjusted VPL

Adjusted HPL

From reports since Jan. 2008

Max. 95% PLs among stations

in CONUS (“InOut” set)

Conservative reduction factors from PAN data:

VPL / 4.0

HPL / 2.5


A Combined “Average/Specific” Risk Approach

• Depending on user and decision maker risk aversion, separate “average risk” and “specific risk” integrity requirements could be issued.– Both apply at all times one or the other will tend to

dominate for a particular application.

• For example: 10-7 integrity risk per operation (“average”) plus requirement that a worst-case undetected condition cannot increase the total vehicle loss risk by more than a factor of 10.– For aircraft case, factor of 10 increase in total risk equates to

specific risk requirement of 10-5 per operation for nav. system (more strictly, 9 10-6)

– Specific factors for each vehicle and application would vary.

– There is no “correct” degree of risk aversion.


Summary

• Existing integrity assurance procedures for SBAS and GBAS are unique to aviation and its history and may not be suitable for other users.

• SBAS (and GBAS) data analysis suggests that 10-7 HPL and VPL can be greatly reduced if “average risk” approach is taken.

– Examination of past data is useful, but more thorough PRA analysis should be conducted.

• If worst-case elements of risk assessment are still desired, an average/specific risk mixture can be used.

– This flexible “mixture” capability should satisfy almost any level of user and decision maker risk aversion.


Backup Slides follow…


WAAS VPE from FAA PAN Data (3rd Qtr 2010: July – Sept.)

Source: WAAS PAN Report #34, Oct. 2010.

http://www.nstb.tc.faa.gov/

DisplayArchive.htm

Max. VPE 7 m at

Barrow, AK

VPE (m)

No

. of

Sam

ple

s

Meas. from 37 WAAS stations







Example Error Table from PAN #34(from PAN #34)


Max. VPE and VPL from WAAS PAN Data (1 Jan. 2004 – 30 Sept. 2010)

PAN Report Inner WRS

Inner VPE_95%

Inner VPE_Max

Inner VPL Outer WRS

Outer VPE_95%

Outer VPE_Max

Outer VPL Remote WRS

Remote VPE_95%

Remote VPE_Max

Remote VPL

8 Chicago 1.086 7.541 49.612 Minneapolis 1.710 9.133 37.430 N/A N/A N/A N/A9 Dallas 1.442 8.191 39.956 Minneapolis 1.695 7.794 40.806 N/A N/A N/A N/A10 Dallas 1.388 8.722 43.829 Minneapolis 1.790 7.376 32.210 N/A N/A N/A N/A11 Dallas 1.371 8.280 31.969 Minneapolis 1.501 8.034 37.367 N/A N/A N/A N/A12 Dallas 1.298 9.301 33.699 Salt Lake City 1.155 8.581 47.939 N/A N/A N/A N/A13 Dallas 1.504 9.457 28.399 Minneapolis 1.765 12.756 44.758 N/A N/A N/A N/A14 Dallas 1.141 6.426 26.887 Oakland 1.706 7.931 37.235 N/A N/A N/A N/A15 Dallas 1.469 6.719 24.612 Minneapolis 1.956 7.439 28.722 N/A N/A N/A N/A16 Albuquerque 0.934 8.195 24.246 Minneapolis 1.157 8.002 31.380 N/A N/A N/A N/A17 Dallas 1.202 7.893 34.771 Oakland 1.273 6.385 47.296 N/A N/A N/A N/A18 Dallas 1.210 6.888 37.435 Oakland 1.228 7.296 46.769 N/A N/A N/A N/A19 Dallas 1.281 6.879 35.097 Miami 1.657 6.913 46.396 Fairbanks 1.080 7.395 40.63220 Dallas 1.184 6.040 30.050 Seattle 0.886 5.858 22.705 Fairbanks 1.062 22.492 33.62021 Dallas 1.028 5.064 26.238 Miami 1.231 5.160 37.664 Kotzebue 1.183 37.308 39.90222 Denver 1.281 3.975 34.868 Oakland 1.043 4.119 30.970 Fairbanks 1.118 9.255 34.79323 Kansas City 0.945 5.016 24.232 Minneapolis 1.067 5.029 32.445 Puerto Vallarta 1.466 5.854 40.93724 Memphis 0.889 4.800 24.742 Seattle 0.801 4.273 20.643 Tapachula 1.917 7.347 44.25925 Denver 0.800 3.401 27.877 Seattle 0.766 4.553 23.230 San Juan 1.300 5.859 31.84226 Denver 1.100 5.025 28.390 Oakland 1.061 4.808 23.802 S.J. Del Cabo 1.138 5.566 31.80627 Denver 1.022 4.571 25.254 Seattle 0.915 4.972 20.294 Iqaluit 2.087 6.977 28.36228 Chicago 0.852 4.046 21.989 Miami 2.041 4.462 28.787 Fairbanks 0.997 8.018 35.47829 Cleveland 1.041 4.664 24.292 Miami 1.537 4.384 29.033 Barrow 1.128 6.733 26.19830 Dallas 1.001 4.459 50.101 Wash DC 1.124 4.589 33.014 Iqaluit 1.731 9.768 42.10331 Denver 1.108 5.045 25.872 Miami 1.612 4.240 24.229 Iqaluit 1.766 7.556 27.88232 Cleveland 1.001 4.143 28.377 Miami 2.005 4.738 26.618 Iqaluit 1.869 8.106 45.03333 Denver 0.938 4.754 36.569 Miami 1.298 4.516 30.514 Barrow 1.245 7.700 38.50034 Memphis 1.048 4.070 13.567 Seattle 0.849 4.920 37.557 Barrow 1.165 6.975 44.427

Ave 1.132 6.058 30.849 1.364 6.232 33.326 1.391 10.182 36.611

Max 1.504 9.457 50.101 2.041 12.756 47.939 2.087 37.308 45.033

(all numbers are in meters)


95% and Max. VPE from FAA PAN Data (1 Jan. 2004 – 30 Sept. 2010)

10 15 20 25 30 350

5

10

15

20

25

30

35

40

45

50


Ver

tica

l P

os

itio

n E

rro

r (m

eter

s)

VAL for LPV

Note: VPL always bounds VPE.

Remote Stations

Outer Stations

Inner Stations

95% VPE

Max. VPE

Severe iono. scintillation in Alaska in March and May 2007 (user receiver should prevent)


WAAS VPE vs. VPL in CONUS (2003 – 2006) (from Wanner, et al, 2006)

Ver

tica

l P

os

itio

n E

rro

r (m

eter

s)

99.99% VPE

99.9% VPE

95% VPE

99% VPE

Mean VPE

1 VPE

Ratios:

VPE99.99

VPL

%

6.9 6.4 6.8 6.5 6.0 6.8 6.8


WAAS Max. VPE in CONUS (2003 – 2006) (from Wanner, et al, 2008)


An “Average Risk” Approach to SBAS (and GBAS) – word version

• Data imply an “average risk” equivalent VPL for WAAS ~ 4 – 5 times lower than current value.

• Re-assess “rare-normal” and faulted error models and data to build a “certifiable” safety case.

– Multiple rare-normal (“fault-free”) models built from existing data to incorporate remaining uncertainty

– All fault-mode analyses follow the same approach:• Estimate prior fault probabilities and probability uncertainties.

• Simulate all significant variations of each fault type rather than “worst case” focus convolve with prior dist. to estimate risk.

– Faults whose impact is driven by worst-case scenarios (ionosphere, signal deformation) will become less important.

– Multiple-fault scenarios neglected as too improbable may become more important, as probabilistic weighting of risk may show that fault-combination cases are non-negligible.


A Combined “Average/Specific” Risk Approach (1)

Derived from FAA “Hazard Risk Model” (1) and Simplified Aircraft Accident Risk Breakdown (2)

(1) FAA System Safety Handbook, 2008. http://www.faa.gov/library/manuals/aviation/risk_management/ss_handbook/

(2) R. Kelly and J. Davis, “Required Navigation Performance (RNP),” Navigation, Spring 1994.

Catastrophic (Likely a/c hull

loss)

Hazardous (Risk of a/c loss; Severe

loss of safety margin)

Major (Slight risk of aircraft loss/pilot challenged)

10-5 10-7 10-9

10-6

Overall a/c loss prob.

Loss prob. due to equipment failure

~ 10% ~ 1% (~ 100 systems)

Loss prob. due to GNSS nav. failure

~ 1%

10-7 10-9

http://www.faa.gov/library/manuals/aviation/risk_management/ss_handbook/

Documents

SBAS and GBAS Integrity for Non- Aviation Users: Moving Away from "Specific Risk" ION ITM 2011San Diego, CA. 25 January 2011 Sam Pullen, Todd Walter, and