Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse

Bahadir K. Gunturk 1

Phase Correlation


Phase Correlation

Take cross correlation

Take inverse Fourier transform

Location of the impulse function gives the translation amount between the images


Phase Correlation

Computer Vision

Stereo Vision


Coordinate Systems

Let O be the origin of a 3D coordinate system spanned by the unit vectors i, j, and k orthogonal to each other.

i

j

kO

Px OP i

��

y OP j��

z OP k��

OP x y z i j k��

x

y

z

PCoordinate vector


Homogeneous Coordinatesn

a

b

c

HH

O

P

x

y

z

P

0HP OH ��

2 2 2( ) 0ax by cz a b c

0ax by cz d

0

1

x

ya b c d

z

Homogeneous coordinates 0 H P

P

TH


Coordinate System Changes

Translation

BPAP



Rotation

where

Exercise: Write the rotation matrix for a 2D coordinate system.

ˆ

ˆ

ˆ

AX B

AY B

AZ B

B i P

B j P

B k P



Rotation + Translation

3 3 3 1

1 3

''

''

0 1 ''

1 1

x xx x

R ty yy R y t

z zz z


Perspective Projection

Perspective projection equations

' ' 'x y z

x y z


Review: Pinhole Camera


Review: Perspective Projection

' ' 'x y f

x y z


Multi-View Geometry

Relates

• 3D World Points

• Camera Centers

• Camera Orientations

• Camera Parameters

• Image Points


Stereo

scene pointscene point

optical centeroptical center

image planeimage plane

p p’


Finding Correspondences

p p’


Three Questions

Correspondence geometry: Given an image point p in the first view, how does this constrain the position of the corresponding point p’ in the second?

Camera geometry (motion): Given a set of corresponding image points {pi ↔ p’i}, i=1,…,n, what are the cameras C and C’ for the two views? Or what is the geometric transformation between the views?

Scene geometry (structure): Given corresponding image points pi ↔ p’i and cameras C, C’, what is the position of the point X in space?


Stereo Constraints

X1

Y1

Z1

O1

Image plane

Focal plane

M

p p’

Y2

X2

Z2O2

Epipolar Line

Epipole


Epipolar Constraint


From Geometry to Algebra

O O’

P

pp’

All vectors shown lie on the same plane.


From Geometry to Algebra

O O’

P

pp’

( , ,1)[ ( ')] 0 with

' ( ', ',1)

T

T

p u vp t Rp

p u v


Matrix form of cross product

20

0

0

y z z y z

z x x z z x

x y y z y x

a b a b a a

a b a b a b a a b a b

a b a b a a

( ) 0

( ) 0

a a b

b a b

a×b=|a||b|sin(η)u a=axi+ayj+azk

b=bxi+byj+bzk


The Essential Matrix

( , ,1)[ ( ')] 0 with

' ( ', ',1)

T

T

p u vp t Rp

p u v

' 0Tp Ep

' 0 with Tp Ep E t R Essential matrix


Stereo Vision

Two cameras. Known camera positions. Recover depth.


Recovering Depth Information

OO22

P’P’22=Q’=Q’22

PP

QQ

OO11

P’P’11Q’Q’11

Depth can be recovered with two images and triangulation.


A Simple Stereo System

Zw=0

LEFT CAMERA

Left image:reference

Right image:target

RIGHT CAMERA

Elevation Zw

disparity

Depth Z

baseline


Stereo View

Left View Right View

Disparity


Stereo Disparity The separation between two matching objects

is called the stereo disparity.


Parallel Cameras

ZT

fZxxTlr

OOll OOrr

PP

ppll pprr

TT

ZZ

xxll xxrr

ff

T is the stereo baseline

rlxx

TfZ

rlxxd Disparity:


Finding Correspondences


Correlation Approach

For Each point (xl, yl) in the left image, define a window centered at the point

(xl, yl)LEFT IMAGE



… search its corresponding point within a search region in the right image

(xl, yl)RIGHT IMAGE



… the disparity (dx, dy) is the displacement when the correlation is maximum

(xl, yl)dx(xr, yr)RIGHT IMAGE


Stereo correspondence

Epipolar Constraint Reduces correspondence problem to 1D search along epipolar lines

epipolar planeepipolar lineepipolar lineepipolar lineepipolar line


For each epipolar lineFor each pixel in the left image

• Compare with every pixel on same epipolar line in right image

• Pick pixel with the minimum matching error

Of course, matching single pixels won’t work; so, we match regions around pixels.

Stereo correspondence


Comparing Windows ==??

ff gg

MostMostpopularpopular

For each window, match to closest window on epipolar line in other image.


Maximize Cross correlation

Minimize Sum of Squared Differences

Comparing Windows


Feature-based correspondence Features most commonly used:

Corners Similarity measured in terms of:

surrounding gray values (SSD, Cross-correlation) location

Edges, Lines Similarity measured in terms of:

orientation contrast coordinates of edge or line’s midpoint length of line


Feature-based Approach

For each feature in the left image…

LEFT IMAGE

corner line

structure


Feature-based Approach

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

RIGHT IMAGE

corner line

structure


Correspondence Difficulties Why is the correspondence problem difficult?

Some points in each image will have no corresponding points in the other image.(1) the cameras might have different fields of view.

(2) due to occlusion.

A stereo system must be able to determine the image parts that should not be matched.


Structured Light Structured lighting

Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult).

Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth.

Finding and matching such points is simplified by knowing the geometry of the projected patterns.


Stereo results

Ground truthScene

Data from University of Tsukuba

(Seitz)


Results with window correlation

Estimated depth of field(a fixed-size window)

Ground truth

(Seitz)


Results with better method

A state of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)


Window size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi,

A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

Effect of window size

(Seitz)


Other constraints

It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t

change too quickly.)


Parameters of a Stereo System Intrinsic Parameters

Characterize the transformation from camera to pixel coordinate systems of each camera

Focal length, image center, aspect ratio

Extrinsic parameters Describe the relative

position and orientation of the two cameras

Rotation matrix R and translation vector T

pl

pr

P

Ol Or

Xl

Xr

Pl Pr

fl fr

Zl

Yl

Zr

Yr

R, T


Applications

courtesy of Sportvision

First-down line


ApplicationsVirtual advertising

courtesy of Princeton Video Image

Documents

Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse