Feature Detection and Matching

7/30/2019 Feature Detection and Matching

1/80

Feature Detection and Matching

Reading: Szeliski Chapter 4

Pages: 208~266

http://staff.ustc.edu.cn/~xjchen99/teaching/teaching.html


2/80

Image Matching

by Diva Sian

by swashford
http://www.flickr.com/photos/diaphanus/136915456/http://www.flickr.com/photos/swashford/428567562/http://www.flickr.com/photos/swashford/428567562/http://www.flickr.com/photos/diaphanus/136915456/


3/80

Harder Case

by Diva Sian by scgbt
http://www.flickr.com/photos/diaphanus/136915456/http://www.flickr.com/photos/scpgt/328570837/http://www.flickr.com/photos/scpgt/328570837/http://www.flickr.com/photos/diaphanus/136915456/


4/80

Even Harder Case

How the Afghan Girl was Identified by Her Iris Patterns Read the story
http://www.cl.cam.ac.uk/~jgd1000/afghan.htmlhttp://www.cl.cam.ac.uk/~jgd1000/afghan.html


5/80

Harder still?

NASA Mars Rover images


6/80

NASA Mars Rover images

with SIFT feature matchesFigure by Noah Snavely

Answer below (look for tiny colored squares)


7/80

Features

All is Vanity, by C. Allan Gilbert, 1873-1929Readings

Szeliski, Ch 4.1

(optional) K. Mikolajczyk, C. Schmid, A performance evaluation of local

descriptors. In PAMI 27(10):1615-1630- http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_
http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_pami2004.pdfhttp://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_pami2004.pdfhttp://upload.wikimedia.org/wikipedia/en/c/c9/Allisvanity.jpg


8/80

Image Matching


9/80

Image Matching


10/80

Invariant Local Features

Find features that are invariant to transformations

geometric invariance: translation, rotation, scale

photometric invariance: brightness, exposure,

Feature Descriptors


11/80

Advantages of Local Features

Locality

features are local, so robust to occlusion and clutter

Distinctiveness:

can differentiate a large database of objects

Quantity

hundreds or thousands in a single image

Efficiency

real-time performance achievableGenerality

exploit different types of features in different situations


12/80

More Motivation

Feature points are used for:

Image alignment (e.g., mosaics)

3D reconstruction

Motion tracking Object recognition

Indexing and database retrieval

Robot navigation

other


13/80

Features

Point/patch,

Edge/curve

Region


14/80

Want Uniqueness

Look for image regions that are unusual

Lead to unambiguous matches in other images

How to define unusual?


15/80


16/80

Corner Detection

Basic idea: Find points where two edges meeti.e., high gradient in twodirections

Cornerness is undefined at a single pixel, because theres only onegradient per point

Look at the gradient behavior over a small window

Categories image windows based on gradient statistics Constant: Little or no brightness change

Edge: Strong brightness change in single direction

Flow: Parallel stripes

Corner/spot: Strong brightness changes in orthogonal directions


17/80

Matching Criterion

(weighted) Summed Square Difference SSD

Compare an image patch against itself


18/80

Local Measures of Uniqueness

Suppose we only consider a small window of pixels

What defines whether a feature is a good or bad candidate?

Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.


19/80

Feature Detection

flat region:

no change in all

directions

edge:

no change along

the edge direction

corner:

significant change

in all directions

Local measure of feature uniqueness

How does the window change when you shift it? Shifting the window in any direction causes a big change

Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.


20/80

Consider shifting the window W by (u,v)

how do the pixels in W change?

compare each pixel before and after by

summing up the squared differences

(SSD) this defines an SSD error ofE(u,v):

Feature Detection: Mathematics

W


21/80

Harris Detector: Mathematics

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Change of intensity for the shift [u,v]:

IntensityShifted

intensityWindowfunction

orWindow function w(x,y) =

Gaussian1 in window, 0 outside

Auto-correlationfunction


22/80

Auto-Correlation Function


23/80

Auto-Correlation Function

Good unique minimum 1D aperture problem No good peak


24/80

Taylor Series expansion of I:

If the motion (u,v) is small, then first order approx is good

Small Motion Assumption


25/80


Image gradient

Harris detector with a [-2,-1,0,1,2] filter for Ix

Gaussian filter


26/80

Plugging this into the formula on

Small Motion Assumption

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y


27/80


W


28/80

Feature Detection: MathematicsThis can be rewritten:

x-

x+

Auto-correlation matrix


29/80


For the example above

You can move the center of the blue window to anywhere on the

blue unit circle

Which directions will result in the largest and smallest E values?

We can find these directions by looking at the eigenvectors ofH


30/80

Quick eigenvalue/eigenvector review

The eigenvectors of a matrix A are the vectors x that satisfy:

The scalar is the eigenvalue corresponding to x

The eigenvalues are found by solving:

In our case, A = H is a 2x2 matrix, so we have

The solution:

Once you know , you find x by solving


31/80

Feature Detection: MathematicsThis can be rewritten:

Eigenvalues and eigenvectors of H

Define shifts with the smallest and largest change (E value)

x+ = direction of largest increase in E.

+ = amount of increase in direction x+

x- = direction of smallest increase in E.

- = amount of increase in direction x+

x-

x+


32/80


( , ) ,u

E u v u v H

v

Intensity change in shifting window: eigenvalue analysis

1, 2eigenvalues ofH

(max)-1/2

(min)-1/2

EllipseE(u,v) = const

If we try every possible orientation n,

the max. change in intensity is 2


33/80


1

2

Corner

1 and 2 are large,

1 ~

2;Eincreases in all

directions

1 and 2 are small;

Eis almost constant

in all directions

Edge

1 >> 2

Edge

2 >> 1

Flat

region

Classification of image

points using eigenvalues

ofM:


34/80

Feature Detection: MathematicsHow are +, x+, -, and x+ relevant for feature detection?

Whats our feature scoring function?


35/80

Feature Detection: MathematicsHow are

+, x

+,

-, and x

+relevant for feature detection?

Whats our feature scoring function?

Want E(u,v) to be large for small shifts in alldirections

the minimum ofE(u,v) should be large, over all unit vectors [u v]

this minimum is given by the smaller eigenvalue (

-) ofH


36/80

Feature DetectionHeres what you do

Compute the gradient at each point in the image

Create the Hmatrix from the entries in the gradient

Compute the eigenvalues.

Find points with large response (- > threshold)

Choose those points where - is a local maximum as features


37/80

Feature DetectionHeres what you do

Compute the gradient at each point in the image

Create the Hmatrix from the entries in the gradient

Compute the eigenvalues.

Find points with large response (- > threshold)

Choose those points where - is a local maximum as features

[Shi and Tomasi 1994]


38/80

Harris Detector

Measure of corner response:

2

det traceR M k M

1 2

1 2

det

trace

M

M

(kempirical constant, k= 0.04-0.06)

Harris and Stephens 1988

The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22 Very similar to - but less expensive (no square root)


39/80

Harris Detector: Mathematics

1

2 Corner

Edge

Edge

Flat

R depends only on eigenvalues

of H

R is large for a corner

R is negative with large

magnitude for an edge

|R| is small for a flat region

R > 0

R < 0

R < 0|R| small


40/80

Harris Detector

The Algorithm:

Find points with large corner response function R

(R > threshold)

Take the points of local maxima ofR


41/80

Harmonic Mean

Smoother function in the region where

Brown, M., Szeliski, R., and Winder, S. (2005).

0 1

0 1

det

trf

H

H

0 1


42/80

Isocontours of Response

Harris Detector: Workflow


43/80




44/80


Compute corner responseR



45/80


Find points with large corner response:R>threshold



46/80


Take only the points of local maxima ofR



47/80



48/80

Example: Gradient Covariances

Full imageDetail of image with gradient covar-

iance ellipses for 3 x 3 windows

from Forsyth & Ponce

Corners are whereboth eigenvalues are big


49/80

Example: Corner Detection

(for camera calibration)

courtesy of B. Wilburn


50/80

Example: Corner Detection

courtesy of S. Smith

SUSAN corners


51/80

Harris Detector: Summary

Average intensity change in direction [u,v] can be expressed as a bilinearform:

Describe a point in terms of eigenvalues ofM:

measure of corner response

A good (corner) point should have a large intensity change in all directions, i.e.R should be large positive

( , ) ,u

E u v u v Mv

2

1 2 1 2R k


52/80

Outline of Feature Detection

1. Compute the horizontal and vertical derivatives of the image Ix and Iy byconvolving the original image with derivatives of Gaussians (Section3.2.3).

2. Compute the three images corresponding to the outer products of thesegradients. (The matrix H is symmetric, so only three entries are needed.)

3. Convolve each of these images with a larger Gaussian.

4. Compute a scalar interest measure using one of the formulas discussedabove.

5. Find local maxima above a certain threshold and report them asdetected feature point locations.

Ad ti N M i l S i


53/80

Adaptive Non-Maximal Suppression

(ANMS)

Local maxima

Response value should be significantly (10%)larger than all of its neighbors within a radius

(r)

(a) Strongest 250 (b) Strongest 500

Ad ti N M i l S i


54/80

Adaptive Non-Maximal Suppression

(ANMS)

(a) Strongest 250 (b) Strongest 500

(c) ANMS 250, r = 24 (d) ANMS 500, r = 16


55/80

Invariance

Suppose you rotate the image by some angle

Will you still pick up the same features?

What if you change the brightness?

Scale?


56/80

Invariance

Repeatability of feature detector:

frequency with which keypoints are detected in one image are found

within (=1.5) pixels of the corresponding location in a transformed

image


57/80

Scale Invariance

Detect features at a

variety of scales

Multiple resolutions ina pyramid

Matching in allpossible levels


58/80

Multi-Scale Oriented Patches


59/80

Scale invariant detection

Suppose youre looking for corners


60/80

Feature Detection

Key idea: find scale that gives local maximum of f

f is a local maximum in both position and scale

Common definition of f: Laplacian

Difference between two Gaussian filtered images

with different sigmas


61/80

Feature Detection

Stable features in both location and scale

Laplacian of Gaussian (LoG)

Difference of Gaussian2 2 2 2

2 2

1 2

1 2

1 2

2 22 2

1 1 12

x y x y

DoG G G e e


62/80

Slide from Tinne Tuytelaars

Lindeberg et al, 1996

Slide from Tinne Tuytelaars

Lindeberg et al., 1996


63/80


64/80


65/80


66/80


67/80


68/80


69/80


70/80

Scale-Space Feature Detection

Empirical number of levels in each octave = 31.6


71/80

SIFT Feature Detection

Will be explained next week

Rotation Invariance and orientation


72/80

Rotation Invariance and orientation

estimation

Dominant orientation

Average gradient within a region around the keypoint(larger window than detection window to make it more

reliable)

Orientation histogram


73/80

Affine Invariance

For wide baseline stereo matching or objectrecognition


74/80

Affine Invariance

For a small enough patch, any continuous image wrapping canbe well approximated by an affine deformation

Eigenvalue analysis for Hessian matrix or auto-correlation:

using the principle axis and ratios to fit the affine coordinateframe1 1 2 2

, , , x x

(Lindeberg and Garding 1997; Baumberg 2000; Mikolajczyk and Schmid 2004; Mikolajczyk,

Tuytelaars, Schmid et al. 2005; Tuytelaars and Mikolajczyk 2007)


75/80

Affine Invariance

Maximally Stable Extremal Region (MSER) Matas, Chum, Urban et al. (2004)

Algorithm

Binary regions by thresholding the image at all possiblegray levels

Region area changes as the threshold changes

D_area/D_t is minimalmaximal stable


76/80

Keypoint Detection

Very active area (Xiao and Shah 2003; Koethe 2003; Carneiro and Jepson 2005; Kenney, Zuliani, and

Manjunath 2005; Bay, Tuytelaars, and Van Gool 2006; Platel, Balmachnova, Floracket al. 2006; Rosten and Drummond 2006

Invariance to transformation such as scaling, rotations,noise, and blur

These experimental results, code, and pointers to the surveyed

papers can be found on their Web site athttp://www.robots.ox.ac.uk/vgg/research/affine/

(Mikolajczyk, Tuytelaars, Schmid et al. 2005)
http://www.robots.ox.ac.uk/vgg/research/affine/http://www.robots.ox.ac.uk/vgg/research/affine/


77/80

4.1.2 Feature Descriptor

TBD


78/80

SIFT Feature

Scale Invariant Feature Transform (SIFT)

Detect features that densely cover the image

over the full range of scales and locations

Distinctive Image Features from Scale-Invariant Keypoints, David G. Lowe

d h


79/80

Points and Patches

Two approaches to finding feature points andtheir correspondences

detect in one, then track

For video sequence application, tracking, stereo

Detect independently, then match

For image stitching, recognition

d h


80/80

Keypoint Detection and Matching

Four steps:

Feature detection

Feature description

Feature matching

Feature tracking

Documents

Feature Detection and Matching