Stereo Vision ECE 847: Digital Image Processing Stan Birchfield Clemson University

Stereo Vision

ECE 847:Digital Image Processing

Stan BirchfieldClemson University

Modeling from multiple views

time

# cameras

photographbinocular stereo

trinocular stereo

multi-baseline stereo

camcorder

human vision

camera dome

two frames ...

...

– Greek for solid

Stereoscope

Invented by Wheatstone in 1838

Modern version

Can you fuse these?

rightleft

No special instrument needed

Just relax your eyes

L

R

Random dot stereogram

invented by Bela Juleszin 1959

http://www.magiceye.com/faq.htm

Autostereogram

Do you see the shark?

http://en.wikipedia.org/wiki/Autostereogram

Can you cross-fuse these?

right leftNote: Cross-fusion is necessary if distance between images

is greater than inter-ocular distance

L

R

R

L

impossible: instead, trickthe brain:

Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba

Human stereo geometry

http://webvision.med.utah.edu/space_perception.html

fixationpoint

corresponding points

aR

aL

disparity

Horopter

• Horopter: surfacewhere disparity is zero

• For round retina,the theoretical horopteris a circle(Vieth-Muller circle)


Cyclopean image


http://bearah718.tripod.com/sitebuildercontent/sitebuilderpictures/cyclops.jpg

Panum’s fusional area (volume)

• Human visual system is only capable of fusing the two images with a narrow range of disparities around fixation point

• This area (volume) is Panum’s fusional area

• Outside this area we get double-vision (diplopia)

http://www.allaboutvision.com/conditions/double-vision.htm

Human visual pathway

photos courtesy California Academy of Science

Cheetah:More accurate

depthestimation

Antelope:larger field

of view

Prey and predator

Stereo geometry for pinhole cameras with flat retinas

C,C’,x,x’ and X are coplanar

Left camera Right camera

world point

center ofprojection

epipolarplane epipolar line for x

epipole

baseline

M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction

an epipolar plane = plane containing baseline (1-D family)

an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)

Epipolar geometry


What if only C,C’,x are known?

Epipolar geometry

epipole

centerof

projection baseline

All points on p project on l and l’M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

Family of planes and lines l and l’ Intersection in e and e’

Epipolar geometry


Example: Converging cameras


e

e’

Example: Forward motion


Epipolar geometry andFundamental matrix

epipolar line

(epipole: intersection of all epipolar lines)

(computed with 20 points)

Epipolar geometry andFundamental matrix

epipolar line

(epipole: intersection of all epipolar lines)

(computed with 50 points)

Fundamental matrix

point in image 1

point in image 2

fundamentalmatrix

Epipolar lines (1)

epipolar line in image 2associated with (x,y) in image 1

Epipolar lines (2)

epipolar line in image 1associated with (x’,y’) in image 2

Computing the fundamental matrix

knownunknown


knownunknown


1. Construct A (nx9) from correspondences2. Compute SVD of A: A = UVT

3. Unit-norm solution of Af=0 is given by vn (the right-most singular vector of A)

4. Reshape f into F1

5. Compute SVD of F1: F1=UFFVFT

6. Set F(3,3)=0 to get F’(The enforces rank(F)=2)

7. Reconstruct F=UFF’VFT

8. Now xTF is epipolar line in image 2,and Fx’ is epipolar line in image 1

(simple for stereo rectification)

Example: Motion parallel to image plane


Example: Motion parallel to image scanlines

Epipoles are at infinity

Scanlines are the epipolar lines

In this case, the images are said to be “rectified”

Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba

Standard (rectified) stereo geometry

pure translation along X-axis


Perspective projection

X

x

X

x


Rectified geometry

xL xR

XL -XR

b

Standard stereo geometry

• disparity is inversely proportional to depth• stereo vision is less useful for distant objects


Binocular rectified stereo

rightleft

disparity map depth discontinuities

epipolarconstraint

Matching scanlines

inte

nsi

ty

L

R

dis

par

ity

lamp

wall

pixel

rightleft

Stereo matching

• Search is limited to epipolar line (1D)• Look for most similar pixel

?


Aggregation

• Use more than one pixel• Assume neighbors have similar

disparities*

– Use correlation window containing pixel

– Allows to use SSD, ZNCC, Census, etc.


Block matching

Dissimilarity measures

Connection between SSD and cross correlation:

Also normalized correlation, rank, census, sampling-insensitive ...

Most common:

More efficient implementation

Key idea: Summation over window is correlation with box filter, which is separable

Running sum improves efficiency even more

Note: w is half-width

Compare intensities pixel-by-pixel

Comparing image regions

I(x,y) I´(x,y)

Sum of Square Differences


Note: SAD is fast approximation (replace square with absolute value)




I(x,y) I´(x,y)


If energy does not change much, then minimizing SSD equals maximizing cross-correlation:




I(x,y) I´(x,y)

Zero-mean Normalized Cross Correlation

Similarity measures




I(x,y) I´(x,y)

Census

Similarity measures

125 126 125

127 128 130

129 132 135

0 0 0

0 1

1 1 1

(Real-time chip from TYZX based on Census)

only compare bit signature


Sampling-InsensitivePixel Dissimilarity

d(xL,xR)

xL xR

d(xL,xR) = min{d(xL,xR) ,d(xR,xL)}Our dissimilarity measure:

[Birchfield & Tomasi 1998]

IL IR

Given: An interval A such that [xL – ½ , xL + ½] _ A, and

[xR – ½ , xR + ½] _ A

Dissimilarity Measure Theorems

If | xL – xR | ≤ ½, then d(xL,xR) = 0

| xL – xR | ≤ ½ iff d(xL,xR) = 0

∩∩

Theorem 1:

Theorem 2:

(when A is convex or concave)

(when A is linear)

Aggregation window sizes

Small windows • disparities similar• more ambiguities• accurate when correct

Large windows • larger disp. variation• more discriminant• often more robust• use shiftable windows to

deal with discontinuities

(Illustration from Pascal Fua)

Occlusions

(Slide from Pascal Fua)

Left-right consistency check

• Search left-to-right, then right-to-left• Retain disparity only if they agree

xL

d

Do minima coincide?

Results: correlation

disparity mapleft

with left-right consistency check

Constraints

• Epipolar – match must lie on epipolar line• Piecewise constancy – neighboring pixels should usually

have same disparity• Piecewise continuity – neighboring pixels should usually

have similar disparity• Disparity – impose allowable range of disparities (Panum’s

fusional area)• Disparity gradient – restricts slope of disparity• Figural continuity – disparity of edges across scanlines• Uniqueness – each pixel has no more than one match

(violated by windows and mirrors)• Ordering – disparity function is monotonic (precludes thin

poles)

Exploiting scene constraints


Ordering constraint

11 22 33 4,54,5 66 11 2,32,3 44 55 66

2211 33 4,54,5 6611

2,32,3

44

55

66

surface slicesurface slice surface as a pathsurface as a path

occlusion right

occlusion left


Uniqueness constraint

• In an image pair each pixel has at most one corresponding pixel– In general one corresponding pixel– In case of occlusion there is none


Disparity constraint

surface slicesurface slice surface as a pathsurface as a path

bounding box

dispa

rity b

and

use reconstructed features to determine bounding box

constantdisparitysurfaces


Figural continuity constraint

right left

[University of Tsukuba]

Cooperative algorithm

Disparity space image

Dynamic Programming: 1D Search

Dis

par

ity

map

occlusion

depthdiscontinuity

RIGHTL

EF

T

c a r t

ca

t 3 2 1 1 12 1 0 1 21 0 1 2 30 1 2 3 4

string editing:

stereo matching:

penalties: mismatch = 1 insertion = 1 deletion = 1

c a t

c a r t

Minimizing a 2D Cost FunctionalMinimize:

disparity

p,q

p,q

d(p, )2D:

GL

OB

AL

dis

par

ity

pixel?

p,q

d(p, )

1D:

E E d(p, ) u(l )data smoothness p p,q p,q

{p,q} N

Global

u(l )p,q

Discontinuity penalty:

lp,q

minimum cut = disparity surface

u(l )= lp,q p,q p,qsolves

LO

CA

L Local (GOOD)

(BAD)

Multiway-Cut:2D Search

pixels

labels

pixels

labels

[Boykov, Veksler, Zabih 1998]

Multiway-Cut Algorithm

),( x'x ))(, x(x fg

minimum cut

),(

)]()()[,())(,x'xx

x'xx'xx(x fffg Minimizes

source label

sink label

pixels

(cost of label discontinuity)

(cost of assigninglabel to pixel)

pixels

labels

Energy minimization


Graph Cut


(general formulation requires multi-way cut!)

(Boykov et al ICCV‘99)

(Roy and Cox ICCV‘98)

Simplified graph cut

Correspondence as Segmentation

• Problem: disparities (fronto-parallel) O()surfaces (slanted) O( 2 n)=> computationally intractable!

• Solution: iteratively determine which labels to use

labelpixels

find affineparametersof regions

multiway-cut(Expectation)

Newton-Raphson(Maximization)

Stereo Results (Dynamic Programming)

Stereo Results (Multiway-Cut)

Stereo Results on Middlebury Database

imag

eB

irch

fiel

dT

om

asi 1

999

Ho

ng

-C

hen

200

4

Untextured regions remain a challenge

Multiway-cutDynamic programming

Results: dynamic programming

disparity map

[Bobick & Intille]

left

Results: multiway cut

disparity mapleft

[Kolmogorov & Zabih]

Results: multiway cut (untextured)

disparity map

Multi-camera configurations

Okutami and Kanade

(illustration from Pascal Fua)


Example: Tsukuba

Tsukuba dataset

Real-time stereo on GPU• Computes Sum-of-Square-Differences (use

pixelshader)• Hardware mip-map generation for aggregation over

window• Trade-off between small and large support window

(Yang and Pollefeys, CVPR2003)

290M disparity hypothesis/sec (Radeon9800pro)e.g. 512x512x36disparities at 30Hz

GPU is great for vision too!

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

Consider all paths that satisfy the constraints

pick best using dynamic programming


Hierarchical stereo matching

Dow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges


Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)


Example: reconstruct image from neighboring images


Stereo matching with general camera configuration


Image pair rectification


Planar rectification

Bring two views Bring two views to standard stereo setupto standard stereo setup

(moves epipole to )(not possible when in/close to image)

~ image size

(calibrated)(calibrated)

Distortion minimization(uncalibrated)



polarrectification

planarrectification

originalimage pair


Stereo camera configurations


More cameras

Multi-baseline stereo

[Okutomi & Kanade]

Documents

Stereo Vision ECE 847: Digital Image Processing Stan Birchfield Clemson University