L2OS threshold optimisation 20 June 2014, PM26

JL Vergely, J. Boutin, P. Spurgeon

ACRI-ST,LOCEAN, ARGANS

RFI/outlier detection

Aim :

• To improve the thresholds of the L2OS processor to be applied on TB measurements in order to remove outliers.

About thresholds :

• should be independent on L1c quality products

Thresholds to be testedtest for outlier detection (dwell test)nsig

test for out of range TB detection (FOV test)Tm_out_of_range_affov Tm_out_of_range_eaffovTm_out_of_range_stokes3_affovTm_out_of_range_stokes3_eaffovTm_out_of_range_stokes4_affovTm_out_of_range_stokes4_eaffov

test for oscillation TB detection (FOV test)Ts_stdTs_std_stokes3Ts_std_stokes4

Other tests : max of iteration

Tests conditions

|X_swath| < 400 km

Coast : 1000 km

PCT_var < 80

-40° < lat < 40°

SSS ref: Coriolis global NRTOA (MyOcean)

Day : 1,2,3,4,5/5/2013, L1C v550

L2OS proc : v600 (CATDS processing chain)

Indicators / SSS quality filter

- chi2P : good TB fit if chi2P high. Chi2P > 0.05 in current processor. Warning : Dg_chi2P in L2OS processor = 1-Chi2P

- SSS error < 1.4 psu- mean(SSS SMOS – SSS Coriolis) and std(SSS

SMOS – SSS Coriolis) - X = (SSS SMOS – SSS Coriolis)/SSS_error. X

should be close to a Gaussian law with mean(X)=0 and std(X)=1. Does not depend on SSS accuracy (close to the ratio between empirical error and theoretical error).

Chi2P and RFI

Chi2P, 5/5/2013, asc

Percentage of RFI contamination : january 2012, asc

nsig : full ocean

Outlier detection.

Dwell test.

TB removed if :

|TBsmos – TBmodel – DA| > nsig.rad_noise

DA = mean dwell correction

Current value : 5

Tested values : 2, 3, 4, 5

nsig full ocean

nsig full ocean

Expected distribution

Centred reduced variable

4 sigmas test

Queue distribution : outliers

nsig full ocean

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 2

nsig full ocean

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 3

nsig full ocean

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 4

nsig full ocean

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 5

nsig full ocean

nsig=2: Many outliers at 4 sigmas

nsig : coast (1000km)

nsig coast

Expected distribution

nsig coast

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 2

nsig coast

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 3

nsig coast

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 4

nsig coast

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Mean and std(X)

nsig = 5

nsig coast

nsig=2 : very biased !!

nsig=2: Many outliers at 4 sigmas

Tm_out_of_range_affov or eaffov (polar X,Y,3,4)

Snapshot removed if at least one TB is an outlier : |TB smos – TB model| > Tm_out_of_range

Problem because the test is applied directly on the TBs and not on the TBs normalised by the radiometric noise

X and Y from short and long integration timeCurrent value : 50 K for affov and 100 K for eaffovTested value : 10, 20, 30, 40 K

Tm_out_of_range_affov full ocean

Tm_out_of_range_affov full ocean

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Tm = 10K

Tm_out_of_range_affov full ocean

No specific filter

Chi2p > 0.05 & sigSSS < 1.35

Tm = 40K

Tm_out_of_range_affov full ocean

Tm_out_of_range_eaffov full ocean

Tm=10K :

Little bit better but lost of accuracy

No significative change

(with Tm_out_of_range_affov = 12)

Tm_out_of_range_stokes3_affov full ocean

Tm=6K :

Little bit better

-> try to work in dual pol mode ?

No significative change

(with Tm_out_of_range_affov/eaffov = 12/18)

No significative change

(with Tm_out_of_range_stokes3_affov = 8)

Tm_out_of_range_stokes3_eaffov full ocean

Tm_out_of_range_stokes4_affov full ocean

No significative change

(with Tm_out_of_range_stokes3_affov/eaffov = 8/16)

Tm_out_of_range_stokes4_eaffov full ocean

No significative change

(with Tm_out_of_range_stokes3_affov/eaffov = 8/16 & Tm_out_of_range_stokes4_affov = 10)

Ts_std thresholds

Snapshot is removed if :rms((TB smos –TB model)/ra) > Ts_std

rms((TB smos –TB model)/ra) is expected to be close to 1 (if OTT well applied)

Current value = 2.5

Ts_std full ocean

Ts_std=1.5 :

Little bit better

Ts_std_stokes3 full ocean

Ts_std=0.5 or 1 :

Too low

Too biased

No significative improvement

Ts_std_stokes4 full ocean

No significative improvement

Ts_std=1 :

Too low

Too inaccurate

Comparison current configuration and configuration without thresholds

Comp current/without thres. full ocean

current conf

Chi2p > 0.05 & sigSSS < 1.35

Comp current/without thres. full ocean

without filterNo signicative change

Chi2p > 0.05 & sigSSS < 1.35

Comp current/without thres. full ocean

A little bit better without thresholds

Comp current/without thres. coast

Without thresholds : better for 4 sigmas SSS

5 day processing

1,2,3,4,5/05/2013; nsig = 2, 3, 4, 5

Dwell test.

TB removed if :

|TBsmos – TBmodel – DA| > nsig.noise

DA = mean dwell correction

Current config : nsig = 5

Without LS mask. chi2P > 0.05nsig=2

nsig=3

nsig=4

nsig=5

Iteration number

2 modes !!

Iteration numberSignature TEC ?

RFI or island ?

Hot spot

Global improvement using iterMax

Small global effect.

What about specific area ?

4 zones with RFI/coast contamination

Pacific

Pacific + coast

Atlantic

Indian ocean

4 zones with RFI/coast contamination

Pacific

Pacific + coast

Atlantic

Indian ocean

iterMax = 5

Itermax=5 is too low : lost of accuracy

Low bias

Gulf of Bengal

iterMax = 10

good accuracy

iterMax = 15

No significative change comparatively to 10

iterMax = 20

iterMax=20 (current)

iterMax=15

iterMax=10

iterMax=5

Stripes

conclusions• Chi2P (or chi2) is improved with TB filtering but SSS biases

increase. • No TB filtering is requiered ? It helps for coast (only from some

threshods) but not for full ocean.• The current configuration has very high thresholds which filter

almost nothing. A configuration with 0 TB filtering gives almost the same results than the current configuration.

• Filtering has to be done at SSS level from Dg_chi2P and sigSSS.

• Use of Tm_out_of_range ? Algorithm to be improved and tested again ?

• Some improvements from St3 filtering : try to work in dual pol ?• Add a specific mask for faint RFI ?• confirmed with more data : tests from 5 day data (for nsig). Use

of iterMax. iterMax = 10 to remove some RFIs/outliers. • Dg_chi2P in L2OS processor = 1-chi2P