CS 556: Computer Vision

Lecture 3

Prof. Sinisa Todorovic

sinisa@eecs.oregonstate.edu

1

Interest Points

• Harris corners

• Hessian points

• SIFT Difference-of-Gaussians

• SURF

2

Properties of Interest Points

• Locality -- robust to occlusion, noise

• Saliency -- rich visual cue

• Stable under affine transforms

• Distinctiveness -- differ across distinct objects

• Efficiency -- easy to compute3

Example of Detecting Harris Corners

4

Harris Corner Detector

5

homogeneous region⇓

no change in all directions

edge⇓

no change along the edge

corner⇓

change in all directions

Source: Frolova, Simakov, Weizmann Institute

scanning window

Harris Corner Detector

6

2D convolutionSource: Frolova, Simakov, Weizmann Institute

E(x, y) = w(x, y) ⇤ [I(x + u, y + v)� I(x, y)]2

Harris Corner Detector

7

change weighting window shifted image original image

Source: Frolova, Simakov, Weizmann Institute

E(x, y) =X

m,n

w(m, n)[I(x + u + m, y + v + n)� I(x + m, y + n)]2

Harris Detector Example

8

input image

E(x, y)I(x, y)

2D map of changes

Harris Corner Detector

9

image derivatives along x and y axes

Taylor series expansion

For small shiftsu � 0v � 0

I(x + u, y + v) ⇡ I(x, y) +@I

@x

u +@I

@y

v

I(x + u, y + v) ⇡ I(x, y) + [Ix

I

y

]

u

v

�

Harris Corner Detector

10

E(x, y) = w(x, y) ⇤✓

I(x, y) + [Ix

I

y

]

u

v

�� I(x, y)

◆2

= w(x, y) ⇤✓

[Ix

I

y

]

u

v

�◆2

= w(x, y) ⇤✓

[Ix

I

y

]

u

v

�◆T ✓[I

x

I

y

]

u

v

�◆

Harris Corner Detector

11

E(x, y) = w(x, y) ⇤ [u v]

I

x

I

y

�[I

x

I

y

]

u

v

�

= [u v]✓

w(x, y) ⇤

I

2x

(x, y) I

x

(x, y)Iy

(x, y)I

x

(x, y)Iy

(x, y) I

2y

(x, y)

�◆

| {z }M(x,y)

u

v

�

Weighted Image Gradient

12

Image is discrete ⇒ Gradient is approximate

We always find the gradient of the kernel !

w(x, y;�) ⇤ I

x

(x, y) = w

x

(x, y;�) ⇤ I(x, y)

w(x, y;�) ⇤ Iy(x, y) = wy(x, y;�) ⇤ I(x, y)

Harris Corner Detector

13

E(x, y) = [u v]M(x, y)

u

v

�

M(x, y) =

(wx

⇤ I)2 (wx

⇤ I)(wy

⇤ I)(w

x

⇤ I)(wy

⇤ I) (wy

⇤ I)2

�

(x,y)

Eigenvalues and Eigenvectors of M

• Eigenvectors of M -- directions of the largest change of E(x,y)

• Eigenvalues of M -- the amount of change along eigenvectors14

��1

��2

E(x, y) = [u v]M(x, y)

u

v

�

Detection of Harris Corners using the Eigenvalues of M

15

�2

�1

Harris Detector

16

f(�) =�1(�)�2(�)

�1(�) + �2(�)=

det(M(�))trace(M(�))

objective function

Harris Detector Example

17

input image

Harris Detector Example

18

f values color codedred = high valuesblue = low values

Harris Detector Example

19

f values thresholded

Harris Detector Example

20

Harris features = Spatial maxima of f values

Example of Detecting Harris Corners

21

22

Hessian determinantHessian determinant

!"

#$%

&=

yyxy

xyxx

IIII

IHessian )(

2))(det( xyyyxx IIIIHessian −=

Ixx

Ixy

Hessian detector (Beaudet, 1978)Hessian detector (Beaudet, 1978)

Iyy

Source: Tuytelaars

Hessian Detector

23

E(x, y) = [u v]H(x, y)

uv

�

H(x, y;�) = w(x, y;�) ⇤

Ixx

(x, y) Ixy

(x, y)Ixy

(x, y) Iyy

(x, y)

�

H(x, y;�) =

wxx

(x, y;�) ⇤ I(x, y) wxy

(x, y;�) ⇤ I(x, y)w

xy

(x, y;�) ⇤ I(x, y) wyy

(x, y;�) ⇤ I(x, y)

�

Hessian Detector

24

objective function

f(�) =�1(�)�2(�)

�1(�) + �2(�)=

det(H(�))trace(H(�))

Properties of Harris Corners

• Invariance to variations of imaging parameters:

• Illumination?

• Camera distance, i.e., scale ?

• Camera viewpoint, i.e., affine transformation?

25

Harris/Hessian Detector is NOT Scale Invariant

26 Source: L. Fei-Fei

“edge”

“edge”“edge”

“corner”

27

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

Source: Tuytelaars

28

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

Source: Tuytelaars

29

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

Source: Tuytelaars

30

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

Source: Tuytelaars

31

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

Source: Tuytelaars

32

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

Source: Tuytelaars

33

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σxIfmii "

�

Source: Tuytelaars

34

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σxIfmii "

�

Source: Tuytelaars

35

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σxIfmii "

�

Source: Tuytelaars

36

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σxIfmii "

�

Source: Tuytelaars

37

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σxIfmii "

�

Source: Tuytelaars

38

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σxIfmii "

�

Source: Tuytelaars

39

Automatic scale selectionAutomatic scale selection

)),((1

σxIfmii�

Function responses for increasing scale Scale trace (signature)

)),((1

σ ""xIfmii� Source: Tuytelaars

Popular Kernels: Difference of Gaussians

40

DoG(�) = G(x, y; k�)�G(x, y;�)

G(x, y;�) =

1p2⇡�

exp

✓�x

2+ y

2

2�

2

◆

Difference ofGaussians

Gaussian

w(�) ⇤ I

Selected Scales = Extrema of DoG/Laplacian

• Convolve the image with Gaussians whose sigma increases

• Then, subsample, and repeat the convolutions

• Finally, find extrema in the 3D DoG or Laplacian space41 Source: D. Lowe

SIFT Detector

• Find a local maximum of

• simultaneously in:

• 2D space of the image

• Scale

42

scale

x

y

← DoG →

← D

oG →

E(x, y;�) = DOG(x, y;�) ⇤ I(x, y)

SURF Detector

43

E(x, y) = [u v]H(x, y)

uv

�

Hessian matrix

H(x, y;�) =

wxx

(x, y;�) ⇤ I(x, y) wxy

(x, y;�) ⇤ I(x, y)w

xy

(x, y;�) ⇤ I(x, y) wyy

(x, y;�) ⇤ I(x, y)

�

implemented using Haar wavelets