Stereo Sebastian Thrun, Gary Bradski, Daniel Russakoff Stanford CS223B Computer Vision (with slides by James Rehg and

Stereo

Sebastian Thrun, Gary Bradski, Daniel RussakoffStanford CS223B Computer Vision

http://robots.stanford.edu/cs223b

(with slides by James Rehg and Zhigang Zhu)

Stereo

Sebastian Thrun Stanford University CS223B Computer Vision

Stereo Vision: Illustration

http://www.well.com/user/jimg/stereo/stereo_list.html


Stereo Vision: Outline

Basic Equations Epipolar Geometry Image Rectification Reconstruction Correspondence (Active Range Imaging Techniques)


Pinhole Camera Model

Imageplane Focal length f

Center ofprojection



Imageplane

),,( ZYXP

),,( ZYXP

f

Oy

x

z

Z

Z

Y

Y

X

X

ZZ

YY

XX

OPPO



Imageplane

),,( ZYXP

),,( ZYXP

f

Oy

x

z

)1,,()1,,(),,(

,

,,

Z

Yf

Z

XfyxZYX

YyXxZ

YfY

Z

XfXfZ


Basic Stereo Derivations

),,(1 ZYXP 1Oy

x

z

f

2Oy

x

z

B

BfxxZ ,,, offunction a as for expression Derive 21

1p

2p


Basic Stereo Derivations

),,(1 ZYXP 1Oy

x

z

f

2Oy

x

z

B

211

11

1

12

1

11 ,

xx

BfZ

Z

Bfx

Z

BXfx

Z

Xfx


What If…?

),,(1 ZYXP 1Oy

x

z

f

2Oy

x

z

B

1p

2p

),,(1 ZYXP 1Oy

x

z

1p

f2O

y

x

z

2p


Epipolar Geometry

pl pr

P

Ol Or

Xl

Xr

Pl Pr

fl fr

Zl

Yl

Zr

Yr

Rrotation Tontranslati


Epipolar Geometry

plp

r

P

Ol Orel er

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles


Epipolar Geometry

Epipolar plane: plane going through point P and the centers of projection (COPs) of the two cameras

Epipoles: The image in one camera of the COP of the other

Epipolar Constraint: Corresponding points must lie on epipolar lines


Essential Matrix

pl pr

P

Ol Orel er

Pl Pr

Orthogonality T, Pl, PlT: 0)( lT

l PTTP

)( TPRP lr Coordinate Transformation:

0

0

0

xy

xz

yz

TT

TT

TT

S

ll SPPT

0)( lT

rT SPPR

0lT

r RSPP

0)( lT

rT PTPRResolves to

RSE Essential Matrix 0lT

r EPP


Essential Matrix

pl pr

P

Ol Orel er

Pl Pr

0

0

0

xy

xz

yz

TT

TT

TT

SRSE Essential Matrix

0 lTr Epp0l

Tr EPP

Projective Line: lr Epu


Fundamental Matrix

Same as Essential Matrix in Camera Pixel Coordinates

0lTr pFp

0lTr Epp

Pixel coordinates 1 lT

r EMMF

Intrinsic parameters


Computing F: The Eight-Point Algorithm

Input: n point correspondences ( n >= 8)– Construct homogeneous system Ax= 0 from

• x = (f11,f12, ,f13, f21,f22,f23 f31,f32, f33) : entries in F• Each correspondence give one equation• A is a nx9 matrix

– Obtain estimate F^ by SVD of A:• x (up to a scale) is column of V corresponding to the least

singular value– Enforce singularity constraint: since Rank (F) = 2

• Compute SVD of F:• Set the smallest singular value to 0: D -> D’• Correct estimate of F :

Output: the estimate of the fundamental matrix F’ Similarly we can compute E given intrinsic

parameters

0lTr pFp

TUDVA

TUDVF ˆ

TVUDF' '


Recitification

Idea: Align Epipolar Lines with Scan Lines.

Question: What type transformation?


Locating the Epipoles

pl pr

P

Ol Orel er

Pl Pr

Input: Fundamental Matrix F– Find the SVD of F– The epipole el is the column of V corresponding to the

null singular value (as shown above)– The epipole er is the column of U corresponding to the

null singular value (similar treatment as for el) Output: Epipole el and er

TUDVF

el lies on all the epipolar lines of the left image

0lTr pFp

0lTr eFp

0leF


Stereo Rectification (see Trucco)

Stereo System with Parallel Optical AxesEpipoles are at infinity

Horizontal epipolar lines

pl

pr

P

Ol Or

Xl

Xr

Pl Pr

Zl

Yl

Zr

Yr

T


pl

pr

P

Ol Or

Pl Pr

Reconstruction (3-D): Idealized


pl

pr

P

Ol Or

Pl Pr

Reconstruction (3-D): Real

See Trucco/Verri, pages 161-171


Correspondence

1P1Oy

x

z

f

2Oy

x

z

1.lp

1,rp

1P

Phantom points


Correspondence via Correlation

Rectified images

Left Right

scanline

SSD error

disparity

(Same as max-correlation / max-cosine for normalized image patch)


Image Normalization

Even when the cameras are identical models, there can be differences in gain and sensitivity.

The cameras do not see exactly the same surfaces, so their overall light levels can differ.

For these reasons and more, it is a good idea to normalize the pixels in each window:

pixel Normalized ),(

),(ˆ

magnitude Window )],([

pixel Average ),(

),(

),(),(

2

),(

),(),(),(

1

yxW

yxWvuyxW

yxWvuyxW

m

mm

m

m

II

IyxIyxI

vuII

vuII


Images as Vectors

Left Right

LwRw

Each window is a vectorin an m2 dimensionalvector space.Normalization makesthem unit length.


Image Metrics

Lw)(dwR

2

),(),(

2SSD

)(

)],(ˆ),(ˆ[)(

dww

vduIvuIdC

RL

yxWvuRL

m

(Normalized) Sum of Squared Differences

Normalized Correlation

cos)(

),(ˆ),(ˆ)(),(),(

NC

dww

vduIvuIdC

RL

yxWvuRL

m

)(maxarg)(minarg2* dwwdwwd RLdRLd


Correspondence Using Correlation

Left Disparity Map

Images courtesy of Point Grey Research


LEFT IMAGE

corner line

structure

Correspondence By Features


Correspondence By Features

RIGHT IMAGE

corner line

structure

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum


Stereo Correspondences

… …Left scanline Right scanline


Stereo Correspondences

… …Left scanline Right scanline

Match

Match

MatchOcclusion Disocclusion


Search Over Correspondences

Three cases:–Sequential – cost of match–Occluded – cost of no match–Disoccluded – cost of no match

Left scanline

Right scanline

Occluded Pixels

Disoccluded Pixels


Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Stereo Matching with Dynamic Programming



Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Start

End



Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal



Correspondence

It is fundamentally ambiguous, even with stereo constraints

Ordering constraint… …and its failure

Figure fromForsyth & Ponce


A Last Word on Correspondences

Correspondens fail for smooth surfaces

There is currently no good solution to the correspondence problem


Summary Stereo Vision

Epipolar Geometry: Corresponding points lie on epipolar line

Essential/Fundamental matrix: Defines this line Eight-Point Algorithm: Recovers Fundamental matrix Rectification: Epipolar lines parallel to scanlines Reconstruction: Minimize quadratic distance Correspondence:

– Minimize Sum of Squares over image correlation– Minimize Sum of Squares of feature characteristics

Many correspondences: Dynamic programming along scanlines (but can fail)


How can We Improve Stereo?

By James Davis, Honda Research


rect

ified

Active Stereo (Structured Light)


Structured Light: 3-D Result

3D Model3D Snapshot

By James Davis, Honda Research


Time of Flight Sensor: Shutter

http://www.3dvsystems.com








Time of Flight Sensor: Scanning





Cleaned up…Raw data

Documents

Stereo Sebastian Thrun, Gary Bradski, Daniel Russakoff Stanford CS223B Computer Vision (with slides by James Rehg and