GRAPH_MATCHING_FOR_EFFICIENT_CLASSIFIERS_ADAPTATION.pdf

The problem of adaptationVQ adaptation

Summary

Graph Matching for Classifier Adaptation

Devis Tuia, Jordi Munoz-Marı and Jesus Malo

Image and Signal Processing GroupUniversity of Valencia, Spain

International Geoscience and Remote Sensing Symposium

Vancouver, 28nd of July 2011

1/24 Devis Tuia


Summary

Acquisition conditions count

During the day, illumination conditions change >> changes in thespectral response of surfaces.

16h 17h

18h 19h

2/24 Devis Tuia


Summary

Acquisition conditions count

These changes can be seen clearly in the RGB space

Sky

100 120 140140

160

180

200

R

G

100 120 140140

160

180

200

R

G

100 120 140140

160

180

200

R

G

100 120 140140

160

180

200

R

G

Veg

etat

ion

0 100 2000

100

200

R

G

0 100 2000

100

200

R

G

0 100 2000

100

200

R

G

0 100 2000

100

200

R

G

Bu

ildin

gs

0 100 2000

100

200

R

G

0 100 2000

100

200

R

G

0 100 2000

100

200

R

G

0 100 2000

100

200

R

G

16h 17h 18h 19h

3/24 Devis Tuia


Summary

Lessons learned

I Shadowing corresponds to a reduction of the intensity of thespectrum, along with a subtle change of chroma.

400 500 600 700 800 9000

200

400

600

800

1000

1200

Wavelength (nm)

DN

Water

Vegetation

Buildings

Roads

Water with shadow

Vegetation with shadow

Buildings with shadow

Roads with shadow

Original image GT Observed spectra

I Illumination produces rotations of the spectral space.It can be catastrophic for a classifier.

I Other effects? ex: vegetation cycles, haze, ...

4/24 Devis Tuia


Summary

Observing the manifoldGraph matchingExperiments

But oftenImage t1 Image t2

I We do not have labeled information on the scene consideredI We have information on scenes at other time instants or taken

by other sensorsI Direct classification can be catastrophic

Source Target OA Kappaimage image µ σ µ σt1 t1 93.46 0.22 0.901 0.003t1 t2 80.67 0.96 0.730 0.013

5/24 Devis Tuia


Summary


How to solve it?

By adaptation(or domain adaptation or transfer learning or color constancy , ....)

Deform the spectrum to match the distributions[useful in change detection or multitemporal]

Deform the classifier to the new data distribution[Domain adaptation, Bruzzone and Marconcini 2009, Gomez-Chova et al 2009]

Look for new training samples, to discover the differentconfigurations

We consider local deformation of the data structure through graphmatching

6/24 Devis Tuia


Summary


How to solve it?


I Deform the spectrum to match the distributions[useful in change detection or multitemporal]

I Deform the classifier to the new data distribution[Domain adaptation, Bruzzone and Marconcini 2009, Gomez-Chova et al 2009]

I Look for new training samples, to discover the differentconfigurations


7/24 Devis Tuia


Summary


How to solve it?


I Deform the spectrum to match the distributions[useful in change detection or multitemporal] >> NOW!

I Deform the classifier to the new data distribution[Domain adaptation, Bruzzone and Marconcini 2009, Gomez-Chova et al 2009]

I Look for new training samples, to discover the differentconfigurations >> Next two talks!


8/24 Devis Tuia


Summary


A multitemporal example

Image t1

G - R - IR space t1

G

R

NIR

01500

1400

2000

Image t2

G - R - IR space t2

G

R

NIR

01500

1400

2000

9/24 Devis Tuia


Summary


Create image representations

Quantization using k-means

0 1500 0

20000

1400

RG

NIR

0 1500 0

20000

1400

RG

NIR

t1 t2

I Slight shifts of distribution

I Rotations and translation

I Local distorsions

10/24 Devis Tuia


Summary


An intuition about good and bad matching

t1 t2 Superposition

I Matching the distributions seems to be the way

11/24 Devis Tuia


Summary


An intuition about good and bad matching

Superposition

(centroids)

Euclidean match

(bad)

Graph match

(good)

I A straight euclidean match can be catastrophic

I Need a more sophisticated matching >> graph matching

12/24 Devis Tuia


Summary


A graph can be described by a series ofI nodes c, the vertices of the graph

c1

c3

c5

c4

c2

I edges (or weights) w , that can be defined by the nearestneighbors rule

c1

c2

c3

c4

c5

W0

1

1

1

1 1

1

1

1

13/24 Devis Tuia


Summary


Graph matching

To match the graphs t1/t2, we displace the nodes at t1 towardsthe nodes at t2. The modified nodes c∗ are defined as

c∗︸︷︷︸New nodes

= minc∗

{ ∑c∈ct1‖ct1 − c∗‖2︸︷︷︸

(1)

+ ‖W t1 −W ∗‖1︸︷︷︸(2)

}

(1) first term minimize nodes displacement>> stay close to the original nodes

(2) second term avoids graph structure changes>> keep the structure among nodes

14/24 Devis Tuia


Summary


Avoid structure changes

1

2

3

4

5

1

3

5

4

2

3

55

1

3

5

4

2

3

1

3

5

4

2

Original t1 Modification 1 Modification 2

W0 W1 W2

1

1

1

1 1

1

1

1

1 1

1 1

1 1

11

11

1

1 1

1

1

1

|| W0 - W1 ||1 = 0 || W0 - W2 ||1 = 4

15/24 Devis Tuia


Summary


Graph matching

I Iterative procedure (node by node in t1)

I Considers a node and its nearest neighbors in t1

I Each node can move towards its nearest neighbors on the t2 graph (blue)

I When stable, move the t1 training points cloud wrt c∗

cc11

c3

c55

cc4

cc22

16/24 Devis Tuia


Summary


Graph matching





c1

c3

c5

c4

cc22

17/24 Devis Tuia


Summary


Graph matching





c1

c3

c5

c4

c2

c

c

c

c

c

18/24 Devis Tuia


Summary


Graph matching





c1

c3

c5 c4

c2

11c

3333c

5c 4

c

c

19/24 Devis Tuia


Summary


Graph matching





cc11

c33

c55 c4

cc22

20/24 Devis Tuia


Summary


Graph matching





Original After matching

21/24 Devis Tuia


Summary


Graph matching result

0 0.1 0.2 0.3 0.4 0.5 0.6 0

0.5

0

0.1

0.2

0.3

0.4

0.5

RG

IR

0 1500 0

20000

1400

RG

NIR

Movements allowed Final local transform

22/24 Devis Tuia


Summary


Results

Model = SVMκ

# training from t1 143 286 1430 2860

Classification of t1 t1 → t1 0.857 0.878 0.901 0.909Classification of t2 t2 → t2 0.794 0.826 0.854 0.860Transfer, no modeladaptation

t1 → t2 0.730 0.747 0.730 0.746

Transfer, withmodel adaptation(k = 50)

t1 → t∗1 → t2 0.761 0.780 0.783 0.788

Transfer, withmodel adaptation(k = 100)

t1 → t∗1 → t2 0.753 0.772 0.792 0.808

I Improvement in all experiments (+0.03 to +0.06 in κ)I Training samples are just translated, they should be rotated as

well (ongoing)I Does not reach performance of t2 since no labeled information

in t2 is added

23/24 Devis Tuia


Summary

Summary

The model proposed

I Enhances adaptation without additional information

I Is based on simple vector quantization based on clustering

I Proved to perform local adaptation maintaining graphstructures

I Next stepsI More elegant cost functionsI Allow smoother movements of the graphI Rotate training pixels with t2 distributionI Going semi-supervised

24/24 Devis Tuia


Summary

Thank you!

http://isp.uv.es/

http://devis.tuia.googlepages.com

[email protected]

25/24 Devis Tuia

Technology

GRAPH_MATCHING_FOR_EFFICIENT_CLASSIFIERS_ADAPTATION.pdf