A methodology for translating positional A methodology for translating positional error into measures of attribute error, and error into measures of attribute error, and combining the two error sourcescombining the two error sources
Yohay Carmel1, Curtis Flather2 and Denis Dean3
1 The Thechnion, Haifa, Israel2 USDA, Forest Service3 Colorado State University
Part 1: bridging the gap between positional error and classification error
classification error -- difference in pixel class between the map and a reference
Positional error(misregistration, location error)
Is the gap between the true location of an item and its location on the
map / image
Positional error may translate to thematic error
positional error largely affects overall thematic error (often more than classification error)(Townshend et al 1992, Dai and Khorram 1998)
Positional error:
RMSE = 2.51 m
classification error:
Accuracy Matrix:
Forest Shrubs GrassForest 221 32 6Shrubs 12 185 18Grass 3 27 149Classified
Reference
Goal 1: find a common denominator for both error typesGoal 2: combine the two error types to get an overall estimate of error (in the context of temporal change)
THE PROBABILITY THAT AN OBSERVED TRANSITION IS CORRECT
positional error affects thematic error
Expressing positional error in terms of thematic error
Shift = 1, 0Shift = 15, 7
Shift = 2, 3
Forest Shrubs GrassForestShrubsGrassClassified
ReferenceExpressing positional error in terms of thematic error
Error model: step 1 (and step 2) of 5
RMSEPositionalError
ClassificationError
Forest Shrubs GrassForest 221 32 6Shrubs 12 185 18Grass 3 27 149
Forest Shrubs GrassForest 24411 3708 1236Shrubs 2627 34763 4172Grass 1700 4944 20240
1 2 3123
Reference
1 2 3123
positional accuracy matrix
ALOC
classification accuracy matrix
ACLASS
Combined
ABOTH
ALOC1,1*ACLASS
2,1 / n+1 1 2 3123
Total n+1 n+2 n+3
ALOC2,1*ACLASS
2,2 / n+2
ALOC3,1*ACLASS
2,3 / n+3
ABOTH2,1
=
+
+
Classified
Classified
Classified
error model: step 3 (of 5)Combining the two accuracy matrices
Forest Shrubs Grass Forest 29652 7458 3668 Shrubs 6450 27885 4885 Grass 3459 5214 15332
Classified
Reference
Error model: step 3 Combined Error Matrix
Error model: step 4
Calculating the combined PCC, and the Combined user accuracy, p(C)
Forest Shrubs Grass User accuracy
Forest 29652 7458 3668 0.73 Shrubs 6450 27885 4885 0.71 Grass 3459 5214 15332 0.64
Classified
Reference
PCC = 0.70
THE PROBABILITY THAT AN OBSERVED STATE IS CORRECT
Error model: step 5
Calculating multi-temporal indices
One such index is The probability that an observed transition is correct
The context of this model is temporal change. The goal is to provide indices for the reliability of an observed change
Example: vegetation changes in Hastings Nature Reserve, California
1939
1956
1971
1995
Example: vegetation changes in Hastings, California
RMSE 1939 = 3.53 m 1995 = 2.51 m
User accuracy for:Grass in 1939 = 0.92Trees in 1995 = 0.91
positional accuracy Classification accuracy
1939 1995
C1 C4
The probability that an observed transition is
correct
p(C1C2…Cn) = p(C1) * p(C2) * … * p(Cn)
C1 C2 C3 C4
1939 1956 1971 1995
Error model: step 5
p(C1C2…Cn) = p(C1) * p(C2) * … * p(Cn)
1939 1956 1971 1995
C1 C2 C3 C4
The probability that an observed transition is correct:
This probability may be calculated as the product of the respective user-accuracy value for the respective
year and class
Example: vegetation changes in Hastings, California
1939 1995
G T
User accuracy for positional Classification Combined
Grass in 1939 0.82 0.92 0.74
Trees in 1995 0.77 0.91 0.72
p(GRASS1939TREE1995) = 0.530.53
p(G1939G1956G1971T1995) = 0.220.221939 1956 1971 1995
G G G T
Transition type
Nature of transition
Proportion in 1939
Probability of being correct, given:
positional error
Classif. error
Combined error
GGGG
Grassland does not change
0.43 0.28 0.69 0.21
CGGG Chaparral burnt in 1955 fire
0.06 0.29 0.60 0.22
All 69 transitions involving forest 0.66 0.26 0.59 0.20
Averaged across the entire study area
1 0.29 0.67 0.22
Indices of accuracy of multi-temporal datasets
(1) errors in each time step are independent of errors in other time steps
(2) positional and classification errors are independent of one another
A simulation study was conducted in order to evaluate how robust is the model in general
and in particular -- to violations of two assumptions:
Maps for simulationsHigh autocorrelation, equal class proportions
Low autocorrelation, unequal class proportions
Int. J. Rem. Sens. 2004
Original map
Spatial error
Classification error
Both error types
SimulationSimulation
Model simulations were conducted under a range of values for:
• Number of map categories (2-4)
• Class Proportions in the original map
• Auto-correlation in the original map
• Auto-correlation in classification error
• Classification error rate
• Positional error rate
• Correlation in error structure between time steps
• Correlation between the two error types
characteristics PCC D
1 2 3 Both
One class dominates 92% 4% 4.00% 0.89 0.08%
One class nearly absent 32% 68% 0.03% 0.77 0.20%
Positional error Large 36% 33% 31.0% 0.56 0.39%
Classification error large 30% 41% 30.0% 0.62 0.25%
Class proportions
Some results of simulation runsN
AA
D
k
i
k
j
bothobs
1 1
||
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Correlation between location and classification errors
Dif
fere
nce
in t
rans
itio
n pr
obab
iliti
es b
etw
een
mod
el
pred
icti
ons
and
sim
ulat
ions
maximum difference average difference
When models assumptions are not met, model fit decreases
Maximum correlation found in real datasets*
*IEEE GRSL 2004
Maximum correlation found in real datasets*
*IEEE GRSL 2004
Transition type
Nature of transition
Proportion in 1939
Probability of being correct, given:
positional error
Classif. error
Combined error
GGGG
Grassland does not change
0.43 0.28 0.69 0.21
CGGG Chaparral burnt in 1955 fire
0.06 0.29 0.60 0.22
All 69 transitions involving forest 0.66 0.26 0.59 0.20
Averaged across the entire study area
1 0.29 0.67 0.22
Indices of accuracy of multi-temporal datasets
PART 2:Controlling data-uncertainty
via aggregation
Pixel size = 0.3 mGrid cell size = 15 m = 2500 pixels
Map aggregation = image degradationoverlay a grid of cells on the image (cell >>pixel)
and define the larger cell as the basic unit
Pixel size = 0.3 mGrid cell size = 15 m = 2500 pixels
31%45%
24%
‘‘soft’ aggregationsoft’ aggregation‘‘hard’ aggregationhard’ aggregation
Map aggregation = image degradationoverlay a grid of cells on the image (cell >>pixel)
and define the larger cell as the basic unit
Impact of positional error is largely reduced when cells are aggregated
ab
c
At the pixel level:
only 55% of the pixels remained unaffected by a minor shift
At the grid cell level:
post-shiftpre-shift0.170.210.340.330.490.46
Impact of positional error is largely reduced when cells are aggregated
This trade-off calls for a model that quantifies the process
to aid decisions on optimal level of aggregation
Aggregation:
Gain in accuracy BUT loss of information
A geometric approach to the impact of positional error
Effective positional error at the grid cell level
2A
eeeAeA yxyx is the proportion of
pixels that transgress into neighboring cells
(RMSE units)
Positional error at the GRID CELL level
p(loc) is the probability that positional error translates into attribute error
pA(loc) is the same probability – in the context of a larger grid cell
)()( locplocp A
The impact of aggregation on thematic accuracy
0.23
A p(loc) cell size error
0.6 m 0.23
6 m 0.14
60 m 0.01
Conclusions
• positional error has a large impact on thematic accuracy, particularly in the context of change
• But can be easily mitigated: increase MMU to >10X[positional error] and do not worry about it.
• Within overall thematic error at the pixel level –classification error component is typically smaller than the positional error component, but is more difficult to get rid of by aggregation.
TODA
THANK YOU