A methodology for translating positional A methodology for translating positional error into measures of attribute error, and error into measures of attribute error, and combining the two error sourcescombining the two error sources

Yohay Carmel1, Curtis Flather2 and Denis Dean3

1 The Thechnion, Haifa, Israel2 USDA, Forest Service3 Colorado State University

Part 1: bridging the gap between positional error and classification error

classification error -- difference in pixel class between the map and a reference

Positional error(misregistration, location error)

Is the gap between the true location of an item and its location on the

map / image

Positional error may translate to thematic error

positional error largely affects overall thematic error (often more than classification error)(Townshend et al 1992, Dai and Khorram 1998)

Positional error:

RMSE = 2.51 m

classification error:

Accuracy Matrix:

Forest Shrubs GrassForest 221 32 6Shrubs 12 185 18Grass 3 27 149Classified

Reference

Goal 1: find a common denominator for both error typesGoal 2: combine the two error types to get an overall estimate of error (in the context of temporal change)

THE PROBABILITY THAT AN OBSERVED TRANSITION IS CORRECT

positional error affects thematic error

Expressing positional error in terms of thematic error

Shift = 1, 0Shift = 15, 7

Shift = 2, 3

Forest Shrubs GrassForestShrubsGrassClassified

ReferenceExpressing positional error in terms of thematic error

Error model: step 1 (and step 2) of 5

RMSEPositionalError

ClassificationError

Forest Shrubs GrassForest 221 32 6Shrubs 12 185 18Grass 3 27 149

Forest Shrubs GrassForest 24411 3708 1236Shrubs 2627 34763 4172Grass 1700 4944 20240

1 2 3123

Reference

1 2 3123

positional accuracy matrix

ALOC

classification accuracy matrix

ACLASS

Combined

ABOTH

ALOC1,1*ACLASS

2,1 / n+1 1 2 3123

Total n+1 n+2 n+3

ALOC2,1*ACLASS

2,2 / n+2

ALOC3,1*ACLASS

2,3 / n+3

ABOTH2,1

=

+

+

Classified

Classified

Classified

error model: step 3 (of 5)Combining the two accuracy matrices

Forest Shrubs Grass Forest 29652 7458 3668 Shrubs 6450 27885 4885 Grass 3459 5214 15332

Classified

Reference

Error model: step 3 Combined Error Matrix

Error model: step 4

Calculating the combined PCC, and the Combined user accuracy, p(C)

Forest Shrubs Grass User accuracy

Forest 29652 7458 3668 0.73 Shrubs 6450 27885 4885 0.71 Grass 3459 5214 15332 0.64

Classified

Reference

PCC = 0.70

THE PROBABILITY THAT AN OBSERVED STATE IS CORRECT

Error model: step 5

Calculating multi-temporal indices

One such index is The probability that an observed transition is correct

The context of this model is temporal change. The goal is to provide indices for the reliability of an observed change

Example: vegetation changes in Hastings Nature Reserve, California

1939

1956

1971

1995

Example: vegetation changes in Hastings, California

RMSE 1939 = 3.53 m 1995 = 2.51 m

User accuracy for:Grass in 1939 = 0.92Trees in 1995 = 0.91

positional accuracy Classification accuracy

1939 1995

C1 C4

The probability that an observed transition is

correct

p(C1C2…Cn) = p(C1) * p(C2) * … * p(Cn)

C1 C2 C3 C4

1939 1956 1971 1995

Error model: step 5

p(C1C2…Cn) = p(C1) * p(C2) * … * p(Cn)

1939 1956 1971 1995

C1 C2 C3 C4

The probability that an observed transition is correct:

This probability may be calculated as the product of the respective user-accuracy value for the respective

year and class

Example: vegetation changes in Hastings, California

1939 1995

G T

User accuracy for positional Classification Combined

Grass in 1939 0.82 0.92 0.74

Trees in 1995 0.77 0.91 0.72

p(GRASS1939TREE1995) = 0.530.53

p(G1939G1956G1971T1995) = 0.220.221939 1956 1971 1995

G G G T

Transition type

Nature of transition

Proportion in 1939

Probability of being correct, given:

positional error

Classif. error

Combined error

GGGG

Grassland does not change

0.43 0.28 0.69 0.21

CGGG Chaparral burnt in 1955 fire

0.06 0.29 0.60 0.22

All 69 transitions involving forest 0.66 0.26 0.59 0.20

Averaged across the entire study area

1 0.29 0.67 0.22

Indices of accuracy of multi-temporal datasets

(1) errors in each time step are independent of errors in other time steps

(2) positional and classification errors are independent of one another

A simulation study was conducted in order to evaluate how robust is the model in general

and in particular -- to violations of two assumptions:

Maps for simulationsHigh autocorrelation, equal class proportions

Low autocorrelation, unequal class proportions

Int. J. Rem. Sens. 2004

Original map

Spatial error

Classification error

Both error types

SimulationSimulation

Model simulations were conducted under a range of values for:

• Number of map categories (2-4)

• Class Proportions in the original map

• Auto-correlation in the original map

• Auto-correlation in classification error

• Classification error rate

• Positional error rate

• Correlation in error structure between time steps

• Correlation between the two error types

characteristics PCC D

1 2 3 Both

One class dominates 92% 4% 4.00% 0.89 0.08%

One class nearly absent 32% 68% 0.03% 0.77 0.20%

Positional error Large 36% 33% 31.0% 0.56 0.39%

Classification error large 30% 41% 30.0% 0.62 0.25%

Class proportions

Some results of simulation runsN

AA

D

k

i

k

j

bothobs

1 1

||

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Correlation between location and classification errors

Dif

fere

nce

in t

rans

itio

n pr

obab

iliti

es b

etw

een

mod

el

pred

icti

ons

and

sim

ulat

ions

maximum difference average difference

When models assumptions are not met, model fit decreases

Maximum correlation found in real datasets*

*IEEE GRSL 2004

Maximum correlation found in real datasets*

*IEEE GRSL 2004

Transition type

Nature of transition

Proportion in 1939

Probability of being correct, given:

positional error

Classif. error

Combined error

GGGG

Grassland does not change

0.43 0.28 0.69 0.21

CGGG Chaparral burnt in 1955 fire

0.06 0.29 0.60 0.22

All 69 transitions involving forest 0.66 0.26 0.59 0.20

Averaged across the entire study area

1 0.29 0.67 0.22

Indices of accuracy of multi-temporal datasets

PART 2:Controlling data-uncertainty

via aggregation

Pixel size = 0.3 mGrid cell size = 15 m = 2500 pixels

Map aggregation = image degradationoverlay a grid of cells on the image (cell >>pixel)

and define the larger cell as the basic unit

Pixel size = 0.3 mGrid cell size = 15 m = 2500 pixels

31%45%

24%

‘‘soft’ aggregationsoft’ aggregation‘‘hard’ aggregationhard’ aggregation

Map aggregation = image degradationoverlay a grid of cells on the image (cell >>pixel)

and define the larger cell as the basic unit

Impact of positional error is largely reduced when cells are aggregated

ab

c

At the pixel level:

only 55% of the pixels remained unaffected by a minor shift

At the grid cell level:

post-shiftpre-shift0.170.210.340.330.490.46

Impact of positional error is largely reduced when cells are aggregated

This trade-off calls for a model that quantifies the process

to aid decisions on optimal level of aggregation

Aggregation:

Gain in accuracy BUT loss of information

A geometric approach to the impact of positional error

Effective positional error at the grid cell level

2A

eeeAeA yxyx is the proportion of

pixels that transgress into neighboring cells

(RMSE units)

Positional error at the GRID CELL level

p(loc) is the probability that positional error translates into attribute error

pA(loc) is the same probability – in the context of a larger grid cell

)()( locplocp A

The impact of aggregation on thematic accuracy

0.23

A p(loc) cell size error

0.6 m 0.23

6 m 0.14

60 m 0.01

Conclusions

• positional error has a large impact on thematic accuracy, particularly in the context of change

• But can be easily mitigated: increase MMU to >10X[positional error] and do not worry about it.

• Within overall thematic error at the pixel level –classification error component is typically smaller than the positional error component, but is more difficult to get rid of by aggregation.

TODA

THANK YOU