On the Interpolation Algorithm Ranking

On the Interpolation Algorithm Ranking

Carlos López-Vázquez

LatinGEO – Lab

SGM+Universidad ORT del Uruguay

10th International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences from 10th to 13th July 2012, Florianópolis, SC, Brazil.

What is algorithm ranking?

There exist many interpolation algorithms

Which is the best? Is there a general answer?

Is there an answer for my particular dataset?

How to define the better-than relation between two given methods?

How confident should I be regarding such answer?

What has been done?

N points sampled somewhere Subdivide N in two sets: Training Set {A} and Test Set {B}

A∩B=Ø; N=#{A}+#{B}

Repeat for all available algorithms:

Define interpolant using {A};

Compare? Typically through RMSE/MAD

Better-Than is equivalent to lower-RMSE

Many papers so far

Permanent interest

How is a typical paper? Takes a dataset as an example

{A} {B}

blindly interpolate at locations of {B}

Compare known values at {B} with those interpolated ones

Is RMSE/MAD/etc. suitable as a metric?

Different interpolation algorithms lead to different look

RMSE might not be representative. Why?

Images from www.spatialanalysisonline.com

Let’s consider spectral properties

Some spectral metric of agreement

For example, ESAM metric

U=fft2d(measured error field), U(i,j)≥0

V=fft2d(interpolated error field), V(i,j)≥0

ideally, U=V

2 2

2( , ) arccos

i ii

i ii i

u vESAM U V

u v

0≤ESAM(U,V)≤1

ESAM(W,W)=1

Hint!: There might be better options than ESAM

How confident should I be regarding such answer?

Given {A} and {B}a deterministic answer

How to attach a confidence level? Or just some uncertainty? Perform Cross Validation (Falivene et al., 2010)

Set #{B}=1, and leave the rest with {A}

N possible choices (events) to select B

Evaluate RMSE for each method and event

Average for each method over N cases

Better-than is now Average-run-better-than

Simulate Sample {A} from N, #{A}=m, m<N

Evaluate RMSE for each method and event, and create rank(i)

Select confidence level, and apply Friedman’s Test to all rank(i)n wines judges each rank k different wines

The experiment

Apply six algorithms

Evaluate RMSE, MAD, ESAM, etc.

Evaluate ranking(i)

Evaluate ranking of means over i

Apply Friedman’s Test and compare

DEM of Montagne Sainte Victoire (France)

Sample {B}, 20 points, held fixed

Do 250 times:

Sample {A} points

Results

Ranking using mean of simulated values might be different from Friedman’s test

Ranking using spectral properties might disagree with that of RMSE/MAD

Friedman’s Test has a sound statistical basis

Spectral properties of the interpolated field might be important for some applications

Questions?

Thank you!

Results

Other results, valid for this particular dataset Ranking using ESAM varies with #{A}

According to ESAM criteria, Inverse Distance Weighting (IDW) quality degrades as #{A} increases

According to RMSE criteria, IDW is the best

With a significative difference w.r.t. the second

With 95% confidence level

Irrespective of #{A}

According to ESAM criteria, IDW is NOT the best

Other possible spectral metrics (to be developed)

0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.01810

7

108

109

1010

1011

1012

1013

1014

Wavenumber [1/m]

Ene

rgy

[m2 ]Results for N=500 data points

ReferenceIDWV4NearestGRIDFIT w/LaplacianGRIDFIT w/smooth=0.5GRIDFIT w/smooth=2