C4 Computer Vision

4 Lectures Michaelmas Term 2003

1 Tutorial Sheet Prof A. Zisserman

Overview

• Lecture 1: Stereo Reconstruction I: epipolar geometry, fundamental matrix.

• Lecture 2: Stereo Reconstruction II: correspondence algorithms, triangulation.

• Lecture 3: Structure and Motion: ambiguities, computing the fundamental matrix, recovering ego-motion, applications.

• Lecture 4: Object detection: the adaBoost algorithm for face detection.

Further reading (www addresses) and the lecture notes are

on http://www.robots.ox.ac.uk/~az/lectures

Stereo Reconstruction

knownknown

cameracamera

viewpointsviewpoints

Shape (3D) from two (or more) images

Example

images

shape

surface

reflectance

Scenarios

The two images can arise from

• A stereo rig consisting of two cameras

• the two images are acquired simultaneously

or

• A single moving camera (static scene)

• the two images are acquired sequentially

The two scenarios are geometrically equivalent

Stereo head

Camera on a mobile vehicle

The objective

Given two images of a scene acquired by known cameras compute the

3D position of the scene (structure recovery)

Basic principle: triangulate from corresponding image points

• Determine 3D point at intersection of two back-projected rays

Corresponding points are images of the same scene point

Triangulation

C C /

The back-projected points generate rays which intersect at the

3D scene point

An algorithm for stereo reconstruction

1. For each point in the first image determine the

corresponding point in the second image

(this is a search problem)

2. For each pair of matched points determine the 3D

point by triangulation

(this is an estimation problem)

The correspondence problem

Given a point x in one image find the corresponding point in the other

image

This appears to be a 2D search problem, but it is reduced to a 1D search

by the epipolar constraint

1. Epipolar geometry

• the geometry of two cameras

• reduces the correspondence problem to a line search

2. Stereo correspondence algorithms

3. Triangulation

Outline

Notation

x x /

X

C C /

The two cameras are P and P/, and a 3D point X is imaged as

for equations involving homogeneous quantities ‘=’ means ‘equal up to scale’

P P/

Warning

P ! "! # $%&'()

! ! #*+,-&.'

" ! "*+,-&.'

Epipolar geometry

Epipolar geometry

Given an image point in one view, where is the corresponding point

in the other view?

epipolar line

?

baseline

• A point in one view “generates” an epipolar line in the other view

• The corresponding point lies on this line

epipole C /C

Epipolar line

Epipolar constraint

• Reduces correspondence problem to 1D search along an

epipolar line

Epipolar geometry continued

Epipolar geometry is a consequence of the coplanarity of the camera

centres and scene point

x x /

X

C C /

The camera centres, corresponding points and scene point lie

in a single plane, known as the epipolar plane

Nomenclature

• The epipolar line l/

is the image of the ray through x

• The epipole e is the point of intersection of the line joining the camera centres

with the image plane

this line is the baseline for a stereo rig, and

the translation vector for a moving camera

• The epipole is the image of the centre of the other camera: e = PC/ , e/ = P/C

xx /

X

C C /

e

left epipolar line

right epipolar line

e/

l/

The epipolar pencil

e e /

baseline

X

As the position of the 3D point X varies, the epipolar planes “rotate” about

the baseline. This family of planes is known as an epipolar pencil. All

epipolar lines intersect at the epipole.

(a pencil is a one parameter family)

Epipolar geometry example I: parallel cameras

Epipolar geometry depends only on the relative pose (position and

orientation) and internal parameters of the two cameras, i.e. the position of

the camera centres and image planes. It does not depend on the scene

structure (3D points external to the camera).

Epipolar geometry example II: converging cameras

Note, epipolar lines are in general not parallel

e e /

Homogeneous notation for lines

• The line l through the two points p and q is l = p x q

Example: compute the point of intersection of the two lines l and m

in the figure below

Proof

y

x

1

2

• The intersection of two lines l and m is the point x = l x m

l

m

which is the point (2,1)

Matrix representation of the vector cross product

Example: compute the cross product of l and m

Algebraic representation of epipolar geometry

We know that the epipolar geometry defines a mapping

x l/

point in first

imageepipolar line in

second image

P

Derivation of the algebraic expression

Outline

Step 1: for a point x in the first image

back project a ray with camera P

Step 2: choose two points on the ray and

project into the second image with camera P/

Step 3: compute the line through the two

image points using the relation l/= p x q

P/

• choose camera matrices

internal

calibrationrotation translation

from world to camera

coordinate frame

• first camera

world coordinate frame aligned with first camera

• second camera

Step 1: for a point x in the first image

back project a ray with cameraP

A point x back projects to a ray

where Z is the point’s depth, since

satisfies

Step 2: choose two points on the ray and

project into the second image with camera P/

P/

Consider two points on the ray

• Z = 0 is the camera centre

• Z = is the point at infinity

Project these two points into the second view

Using the identity

Compute the line through the points

F

F is the fundamental matrix

Step 3: compute the line through the two

image points using the relation l/= p x q

Example I: compute the fundamental matrix for a parallel camera stereo rig

• reduces to y = y/ , i.e. raster correspondence (horizontal scan-lines)

f

f

XY

Z

f

f

XY

Z

Geometric interpretation ?

Example II: compute F for a forward translating camera

f

f

XY

Z

f

f

X Y

Z

first image second image

Summary: Properties of the Fundamental matrix

Stereo correspondence

algorithms

Problem statement

Given: two images and their associated cameras compute

corresponding image points.

Algorithms may be classified into two types:

1. Dense: compute a correspondence at every pixel

2. Sparse: compute correspondences only for features

The methods may be top down or bottom up

Top down matching

1. Group model (house, windows, etc) independently in

each image

2. Match points (vertices) between images

Bottom up matching

• epipolar geometry reduces the correspondence search from 2D

to a 1D search on corresponding epipolar lines

• 1D correspondence problem

b/

a/

bca

CBA

c/

cross-eye viewing random dot stereogram

Correspondence algorithms

Algorithms may be top down or bottom up – random dot stereograms

are an existence proof that bottom up algorithms are possible

From here on only consider bottom up algorithms

Algorithms may be classified into two types:

1. Dense: compute a correspondence at every pixel

2. Sparse: compute correspondences only for features

Dense correspondence algorithm

Search problem (geometric constraint): for each point in the left image, the

corresponding point in the right image lies on the epipolar line (1D ambiguity)

Disambiguating assumption (photometric constraint): the intensity

neighbourhood of corresponding points are similar across images

Measure similarity of neighbourhood intensity by cross-correlation

Parallel camera example – epipolar lines are corresponding rasters

epipolar

line

Intensity profiles

• Clear correspondence between intensities, but also noise and ambiguity

region A

Normalized Cross Correlation

region B

vector a vector b

write regions as vectors

a

b

Cross-correlation of neighbourhood regions

epipolar

line

translate so that mean is zero

(exercise)

left image band

right image band

cross

correlation

1

0

0.5

x

left image band

right image band

cross

correlation

1

0

x

0.5

target region

Why is cross-correlation such a poor measure in the second case?

1. The neighbourhood region does not have a “distinctive” spatial intensity

distribution

2. Foreshortening effects

fronto-parallel surface

imaged length the same

slanting surface

imaged lengths differ

Sketch of a dense correspondence algorithm

For each pixel in the left image

• compute the neighbourhood cross correlation along the corresponding epipolar line in the right image

• the corresponding pixel is the one with the highest cross correlation

Parameters

• size (scale) of neighbourhood

• search disparity

Other constraints

• uniqueness

• ordering

• smoothness of disparity field

Applicability

• textured scene, largely fronto-parallel

Example dense correspondence algorithm

left image right image

right image depth map

3D reconstruction

intensity = depth

Views of a texture mapped 3D triangulation

range map

Pentagon example

left image right image

Rectification

e e /

For converging cameras

• epipolar lines are not parallel

Project images onto plane parallel to baseline

epipolar plane

Rectification continued

Convert converging cameras to parallel camera

geometry by an image mapping

Image mapping is a 2D homography (projective transformation)

(exercise)

Exampleoriginal stereo pair

rectified stereo pair

Note

• image movement (disparity) is inversely proportional to depth Z

• depth is inversely proportional to disparity

Example: depth and disparity for a parallel camera stereo rig

Then, y/ = y, and the disparity

Derivation

x

x/

d

%/ !!"" #! 0

$%

%

&1

'

(

%"

&1'2 $%

(

%"

&1

%

&2$%

(

# 1 %"# %1&$%

(

Triangulation

Problem statement

Given: corresponding measured (i.e. noisy) points x and x/, and

cameras (exact) P and P/, compute the 3D point X

Problem: in the presence of noise, back projected rays do not intersect

C C /

rays are skew in space

Measured points do not lie on corresponding epipolar lines

x x /

1. Vector solution

C C /

Compute the mid-point of the shortest line between the

two rays

2. Linear triangulation (algebraic solution)

Problem: does not minimize anything meaningful

Advantage: extends to more than two views

3. Minimizing a geometric/statistical error

• It can be shown that if the measurement noise is

Gaussian mean zero, , then minimizing

geometric error is the Maximum Likelihood Estimate of X

• The minimization appears to be over three parameters

(the position X), but the problem can be reduced to a

minimization over one parameter

Different formulation of the problem

Minimization method

• Parametrize the pencil of epipolar lines in the first image by t,

such that the epipolar line is l(t)

• Using F compute the corresponding epipolar line in the second

image l/ (t)

• Express the distance function explicitly as a

function of t

• Find the value of t that minimizes the distance function

• Solution is a 6th degree polynomial in t