Special Topic on Image Retrieval

Local Feature Matching Verification

Geometric Verification

• Motivation– Remove false matches by checking geometric

consistency

2

Red line: geometric consistent matchBlue line: geometric inconsistent match

Global Verification: RANSAC

• Take RANSAC as an example– Check geometric consistency from matched feature pairs.

2

1

43

21

t

t

v

u

mm

mm

y

x

Random sampling

Local Geometric-Verification

• Locally nearest neighbors （ Video Goole, cvpr’03）– Matched regions should have a similar spatial layout.– For each match define its search area– Region in the search area that also matches casts a vote

for the image– Reject matches with no support

• Drawback– Sensitive to clutter

Hamming Embedding (ECCV’08)• Introduced as an extension of BOV [Jegou 08]

– Combination of– A partitioning technique (k-means)– A binary code that refine the descriptor

• Representation of a descriptor x– Vector-quantized to q(x) as in standard BOV– short binary vector b(x) for an additional localization in the Voronoi cell

• Two descriptors x and y match iif

Hamming Embedding

• Binary signature generation– Off-line learning

• Random matrix generation• Descriptor projection and assignment• Median values of projected descriptors

– On-line binarization• Quantization assignment• Descriptor projection• Computing the signature:

Local Geometric-Verification

• Bundled feature (CVPR’09)– Group local features in local MSER region.– Increase discriminative power of visual words.– Allowed to have large overlap error.

• Bundle comparison:– Mm(q; p): number of common visual words between two bundles

– Mg(q; p): inconsistency of geometric order in x- and y- direction.

• Drawbacks: Infeasible for rotated bundles.

– Visual words are bundled in MSER regions.– Spatial consistency for bundled features is utilized to weight visual

words. ( ; ) ( ; ) ( ; )m gM q p M q p M q p

Z. Wu, J. Sun, and Q. Ke, “Bundling Features for Large Scale Partial-Duplicate Web Image Search,” CVPR 09

),( pqMvv idftf

# of shared visual words

Spatial consistency

– Great performance for partial-dup detection in over 1 M database– Drawbacks: Infeasible for rotated bundles.

Local Geometric-VerificationBundled feature (CVPR’09)

Global Verification: RANSAC RANSAC: remove outliers by inlier classification

Inliers: true matched features Outliers: false matched features

Assumption of RANdom SAmple Consensus (RANSAC) The original data consists of inliers and outliers. A subset of inliers can estimate a model to optimally explain the inliers.

Estimate the affine transformation by RANSAC

Procedure: Iteratively select a random subset as hypothetical inliers 1. A model is fitted to the hypothetical inliers.2. All other data are tested against the fitted model for inlier classification.3. The model is re-estimated from all hypothetical inliers.4. The model is evaluated by estimating the error of the inliers relative to the model.

Drawbacks: Computationally expensive, not scalableFischler, et al., RANdom SAmple Consensus: a paradigm for model fitting with applications to image analysis and automated

cartography, Comm. of the ACM, 24:381-395, 1981

2

1

43

21

t

t

v

u

mm

mm

y

x

Spatial Coding for Geometric Verification (ACM MM’ 10)

• Motivation– Encode local features’ relative positions into compact binary maps– Check spatial consistency of local matches for geometric verification

• Spatial coding maps– Relative spatial positions between local features.– Very efficient and high precision

1000

1100

1110

1111

1000

1111

1011

1001

Ymap

Xmap

Zhou & Tian, Spatial Coding for large scale partial-duplicate image search. ACM Multimedia 2010.

11

Spatial Map Generation

Rotate 45 degree counterclockwise

In previous case, each quadrant has one part Consider each quadrant is uniformly divided into two parts.

=

12

Spatial Map Generation Generalized spatial map: GX and GY

Each quadrant is uniformly divided into r parts.

… …

r

k

2

k=r-1

k=1

k=0 X-map

X-map

X-map

Y-map

Y-map

Y-map

GX GY

13

Generalized Spatial Coding

1,,2,1,0 ,2

rkr

k

i

iki

ki

y

x

y

x)cos()sin(

)sin()cos(

k

jki

kj

ki

xx

xxkjiGX

if1

if0),,(

k

jki

kj

ki

yy

yykjiGY

if1

if0),,(

Spatial coding maps: Each quadrant uniformly divided into r parts. Decompose the division into r sub-division. Rotate each sub-division to align the axis.

New feature locations after rotation ：

Generalized spatial maps：

14

Spatial Verification Verification with spatial maps GX and GY

Compare the spatial maps of matched features:

k=0, …, r-1; i, j=1, …, N; N: number of matched features Find and delete the most inconsistent matched pair,

recursively:

),,(),,(),,(

),,(),,(),,(

kjiGYkjiGYkjiV

kjiGXkjiGXkjiV

mqy

mqx

1

0 1

),,()(r

k

N

jxx kjiViS

1

0 1

),,()(r

k

N

jyy kjiViS

)(max arg* iSi xi

)(max arg* jSj yj

Vx: inconsistent degree in X-map

Vy: inconsistent degree in Y-map

Identify i* and remove

15

,

01111

10000

10000

10000

10000

,

01011

10000

00000

10000

10000

x

y

2

4

3

1

2

43

1

5

5

11111

01000

01100

01110

01111

10000

11000

11100

11110

11111

10000

11000

11111

11011

11001

11011

01000

11111

01011

01001

YmYqVy

XmXqVx

4

1

1

1

1

3

1

0

1

1

Sy

Sx

,

01111

10000

10000

10000

10000

,

01011

10000

00000

10000

10000

11111

01000

01100

01110

01111

10000

11000

11100

11110

11111

10000

11000

11111

11011

11001

11011

01000

11111

01011

01001

YmYqVy

XmXqVx

4

1

1

1

1

3

1

0

1

1

Sy

Sx

Geometric Verification with Coding Maps

16

SUM

x

y 2

4

3

1

5

2

4

3

1

5

17

Image Plane Division (TOMCCAP’ 10)

2

1

3

5

4

(a)

2

1

3

5

4

(d)

2

1

3

5

4

(c)

2

1

3

5

4

(b)

2

1

3

5

4

(e)

2

1

3

5

4

(f)

18

Geometric Square Coding

• Coordinate adjustment

• Square coding map

Mjiy

x

y

x

j

j

ii

iiij

ij

,0 ,

)cos()sin(

)sin()cos()(

)(

otherwise

syy,xxjiSmap i

ii

ij

ii

ij

0

max if1),(

)()()()(

i

ii

ij

ii

ij

s

yy,xxjiGS

)max(),(

)()()()( Generalized map：

21

3

5

4

2

1

3

5

4

19

Geometric Fan Coding

• Fan coding maps

• Coordinate adjustment

• Generalized coding maps

)()(

)()(

if1

if0),( i

ii

j

ii

ij

xx

xxjiHmap

ii

ij

ii

ij

yy

yyjiVmap

if1

if0),(

j

j

ki

ki

ki

ki

kij

kij

y

x

y

x

)cos()sin(

)sin()cos()()(

)()(

),(

),(

),(),(

),(),(

if1

if0),,( ki

iki

j

kii

kij

xx

xxkjiGH

),(),(

),(),(

if1

if0),,( ki

iki

j

kii

kij

yy

yykjiGV

Geometric Verification

• Compare the fan coding maps of matched features:

• Inconsistency measurement from geometric fan coding:

• Inconsistency measurement from geometric square coding:

• Inconsistency matrix: