Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation...

Bahadir K. Gunturk 1

Phase Correlation

Bahadir K. Gunturk 2

Phase Correlation

Take cross correlation

Take inverse Fourier transform

Location of the impulse function gives the translation amount between the images

Bahadir K. Gunturk 3

Phase Correlation

Computer Vision

Stereo Vision

Bahadir K. Gunturk 5

Coordinate Systems

Let O be the origin of a 3D coordinate system spanned by the unit vectors i, j, and k orthogonal to each other.

i

j

kO

Px OP i

��������������

y OP j��������������

z OP k��������������

OP x y z i j k��������������

x

y

z

PCoordinate vector

Bahadir K. Gunturk 6

Homogeneous Coordinatesn

a

b

c

HH

O

P

x

y

z

P

0HP OH ����������������������������

2 2 2( ) 0ax by cz a b c

0ax by cz d

0

1

x

ya b c d

z

Homogeneous coordinates 0 H P

P

TH

Bahadir K. Gunturk 7

Coordinate System Changes

Translation

BPAP

Bahadir K. Gunturk 8

Coordinate System Changes

Rotation

where

Exercise: Write the rotation matrix for a 2D coordinate system.

ˆ

ˆ

ˆ

AX B

AY B

AZ B

B i P

B j P

B k P

Bahadir K. Gunturk 9

Coordinate System Changes

Rotation + Translation

3 3 3 1

1 3

''

''

0 1 ''

1 1

x xx x

R ty yy R y t

z zz z

Bahadir K. Gunturk 10

Perspective Projection

Perspective projection equations

' ' 'x y z

x y z

Bahadir K. Gunturk 11

Review: Pinhole Camera

Bahadir K. Gunturk 12

Review: Perspective Projection

' ' 'x y f

x y z

Bahadir K. Gunturk 13

Multi-View Geometry

Relates

• 3D World Points

• Camera Centers

• Camera Orientations

• Camera Parameters

• Image Points

Bahadir K. Gunturk 14

Stereo

scene pointscene point

optical centeroptical center

image planeimage plane

p p’

Bahadir K. Gunturk 15

Finding Correspondences

p p’

Bahadir K. Gunturk 16

Three Questions

Correspondence geometry: Given an image point p in the first view, how does this constrain the position of the corresponding point p’ in the second?

Camera geometry (motion): Given a set of corresponding image points {pi ↔ p’i}, i=1,…,n, what are the cameras C and C’ for the two views? Or what is the geometric transformation between the views?

Scene geometry (structure): Given corresponding image points pi ↔ p’i and cameras C, C’, what is the position of the point X in space?

Bahadir K. Gunturk 17

Stereo Constraints

X1

Y1

Z1

O1

Image plane

Focal plane

M

p p’

Y2

X2

Z2O2

Epipolar Line

Epipole

Bahadir K. Gunturk 18

Epipolar Constraint

Bahadir K. Gunturk 19

From Geometry to Algebra

O O’

P

pp’

All vectors shown lie on the same plane.

Bahadir K. Gunturk 20

From Geometry to Algebra

O O’

P

pp’

( , ,1)[ ( ')] 0 with

' ( ', ',1)

T

T

p u vp t Rp

p u v

Bahadir K. Gunturk 21

Matrix form of cross product

20

0

0

y z z y z

z x x z z x

x y y z y x

a b a b a a

a b a b a b a a b a b

a b a b a a

( ) 0

( ) 0

a a b

b a b

a×b=|a||b|sin(η)u a=axi+ayj+azk

b=bxi+byj+bzk

Bahadir K. Gunturk 22

The Essential Matrix

( , ,1)[ ( ')] 0 with

' ( ', ',1)

T

T

p u vp t Rp

p u v

' 0Tp Ep

' 0 with Tp Ep E t R Essential matrix

Bahadir K. Gunturk 23

Stereo Vision

Two cameras. Known camera positions. Recover depth.

Bahadir K. Gunturk 24

Recovering Depth Information

OO22

P’P’22=Q’=Q’22

PP

QQ

OO11

P’P’11Q’Q’11

Depth can be recovered with two images and triangulation.

Bahadir K. Gunturk 25

A Simple Stereo System

Zw=0

LEFT CAMERA

Left image:reference

Right image:target

RIGHT CAMERA

Elevation Zw

disparity

Depth Z

baseline

Bahadir K. Gunturk 26

Stereo View

Left View Right View

Disparity

Bahadir K. Gunturk 27

Stereo Disparity The separation between two matching objects

is called the stereo disparity.

Bahadir K. Gunturk 28

Parallel Cameras

ZT

fZxxTlr

OOll OOrr

PP

ppll pprr

TT

ZZ

xxll xxrr

ff

T is the stereo baseline

rlxx

TfZ

rlxxd Disparity:

Bahadir K. Gunturk 31

Finding Correspondences

Bahadir K. Gunturk 32

Correlation Approach

For Each point (xl, yl) in the left image, define a window centered at the point

(xl, yl)LEFT IMAGE

Bahadir K. Gunturk 33

Correlation Approach

… search its corresponding point within a search region in the right image

(xl, yl)RIGHT IMAGE

Bahadir K. Gunturk 34

Correlation Approach

… the disparity (dx, dy) is the displacement when the correlation is maximum

(xl, yl)dx(xr, yr)RIGHT IMAGE

Bahadir K. Gunturk 35

Stereo correspondence

Epipolar Constraint Reduces correspondence problem to 1D search along epipolar lines

epipolar planeepipolar lineepipolar lineepipolar lineepipolar line

Bahadir K. Gunturk 36

For each epipolar lineFor each pixel in the left image

• Compare with every pixel on same epipolar line in right image

• Pick pixel with the minimum matching error

Of course, matching single pixels won’t work; so, we match regions around pixels.

Stereo correspondence

Bahadir K. Gunturk 37

Comparing Windows ==??

ff gg

MostMostpopularpopular

For each window, match to closest window on epipolar line in other image.

Bahadir K. Gunturk 38

Maximize Cross correlation

Minimize Sum of Squared Differences

Comparing Windows

Bahadir K. Gunturk 39

Feature-based correspondence Features most commonly used:

Corners Similarity measured in terms of:

surrounding gray values (SSD, Cross-correlation) location

Edges, Lines Similarity measured in terms of:

orientation contrast coordinates of edge or line’s midpoint length of line

Bahadir K. Gunturk 40

Feature-based Approach

For each feature in the left image…

LEFT IMAGE

corner line

structure

Bahadir K. Gunturk 41

Feature-based Approach

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

RIGHT IMAGE

corner line

structure

Bahadir K. Gunturk 42

Correspondence Difficulties Why is the correspondence problem difficult?

Some points in each image will have no corresponding points in the other image.(1) the cameras might have different fields of view.

(2) due to occlusion.

A stereo system must be able to determine the image parts that should not be matched.

Bahadir K. Gunturk 43

Structured Light Structured lighting

Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult).

Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth.

Finding and matching such points is simplified by knowing the geometry of the projected patterns.

Bahadir K. Gunturk 44

Stereo results

Ground truthScene

Data from University of Tsukuba

(Seitz)

Bahadir K. Gunturk 45

Results with window correlation

Estimated depth of field(a fixed-size window)

Ground truth

(Seitz)

Bahadir K. Gunturk 46

Results with better method

A state of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)

Bahadir K. Gunturk 47

Window size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi,

A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

Effect of window size

(Seitz)

Bahadir K. Gunturk 48

Other constraints

It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t

change too quickly.)

Bahadir K. Gunturk 49

Parameters of a Stereo System Intrinsic Parameters

Characterize the transformation from camera to pixel coordinate systems of each camera

Focal length, image center, aspect ratio

Extrinsic parameters Describe the relative

position and orientation of the two cameras

Rotation matrix R and translation vector T

pl

pr

P

Ol Or

Xl

Xr

Pl Pr

fl fr

Zl

Yl

Zr

Yr

R, T

Bahadir K. Gunturk 50

Applications

courtesy of Sportvision

First-down line

Bahadir K. Gunturk 51

ApplicationsVirtual advertising

courtesy of Princeton Video Image