1

CSE152, Winter2020

Stereo Introduction to Computer Vision

CSE 152 Lecture 6

CSE152, Winter2020

Announcements •  HW1 assigned, due Thursday at midnight

2

CSE152, Winter 2020

Stereo Vision

CSE152, Winter 2020

Stereo Vision Outline •  Offline:

Calibrate cameras & determine epipolar geometry •  Online

1.  Acquire stereo images 2.  Rectify images to convenient epipolar geometry 3.  Establish correspondence 4.  Estimate depth A

B

C

D

3

CSE152, Winter 2020

Binocular Stereo: Estimating Depth 2-D world with 1-D image plane

Two measurements: xL, xR Two unknowns: X, Z Two constants: Baseline: d Focal length: f Solve for unknowns Disparity: (xL - xR)

(Adapted from Hager)

Z

X (0,0) (d,0)

Z=f

(X,Z)

X =dfxLxL − xR

xL xR

xL = fXZ

xR = fX − dZ

Z = dfxL − xR

{L} {R}

CSE152, Winter 2020

Depth is inverted related to disparity

Faraway points – small disparity Infinitely far, zero disparity

Nearby points – large disparity

Z = dfxL − xR

Disparity: (xL - xR)

4

CSE152, Winter 2020

Binocular Stereo: 3-D world/2-D image: adds some complexity

Given two image measurements p and p’and camera matrices M and M’, estimate P

M’

The issue: The rays pO and p’O’ generally don’t intersect.

M

CSE152, Winter 2020

•  Linear Method: Solve for P as point that closest to and in 3D

•  Non-Linear Method: Find Q by minimizing the image plane distance where q=MQ and q’=M’Q

Given two image measurements p and p’and camera matrices M and M’, estimate P

M M’

Binocular Stereo: 3-D world/2-D image: adds some complexity

Op! "!

O ' p! "!!!

'

5

CSE152, Winter 2020

The need for corresondence

CSE152, Winter 2020

Matching complexity (naïve)

For each point in left mage, there are O(n2) possible matching points in right image.

With n2 pixels in left image, complexity of matching is O(n4)

Can we do better?

Input: two images that are n x n pixels For a given point in the left image, where do we look in the right image?

6

CSE152, Winter 2020

Stereo Vision Outline •  Offline:

Calibrate cameras & determine epipolar geometry •  Online

1.  Acquire stereo images 2.  Rectify images to convenient epipolar geometry 3.  Establish correspondence 4.  Estimate depth A

B

C

D

CSE152, Winter 2020

Epipolar matching

O

p c

O’

c’

p1’ p2’ p’

L’ L

P1

P2 P

e e’ Baseline

•  Potential matches for p have to lie on the corresponding epipolar line L’

•  Epipolar line L’ passes through epipole e’, the intersection of the baseline with the image plane

•  Potential matches for p’ have to lie on the corresponding epipolar line L

7

CSE152, Winter 2020

Epipolar Geometry

•  Epipolar Plane: Any plane that contains the baseline •  Epipoles (e,e’): Two intersection points of baseline with image planes

•  Epipolar Lines (l, l’): Pair of lines from intersection of an epipolar plane with the two image planes

•  Baseline: line connecting two centers of projection O and O’

Epipolar Geometry Terminology

CSE152, Winter 2020

Family of Epipolar Planes

•  Epipolar Plane: Any plane that contains the baseline •  The set of epipolar planes is a family of all planes

passing through the baseline and can be parameterized by the angle about baseline

O O’

8

CSE152, Winter 2020

Epipolar matching

O

p c

O’

c’

p1’ p2’ p’

L’ L

P1

P2 P

e e’ Baseline

•  Potential matches for p have to lie on the corresponding epipolar line L’

•  Epipolar line L’ passes through epipole e’, the intersection of the baseline with the image plane

•  Potential matches for p’ have to lie on the corresponding epipolar line L

CSE152, Winter 2020

Epipolar Constraint Epipolar matching complexity

Using epipolar matching, complexity is reduced from O(n4) to O(n3). Why? •  There are n2 points in the left image •  For each point in the left image, all candidate matches are on an epipolar

line in the right image, and the length of the epipolar line is O(n) •  Therefore, match complexity is O(n2*n)=O(n3)

O

p c

O’

c’

p1’ p2’ p’

L’ L

P1

P2 P

e e’ Baseline

9

CSE152, Winter 2020

Interlude: Skew Symmetric Matrix & Cross Product

•  The cross product a x b of two vectors a=[a1 a2 a3]T and b=[b1 b2 b3] can be expressed a matrix vector product [ax]b where[ax] is the 3x3 skew symmetric matrix:

•  A matrix S is skew symmetric iff S = -ST •  The determinant of a skew symmetric matrix is 0. €

a×[ ] =0 −a3 a2a3 0 −a1−a2 a1 0

$

%

& & &

'

(

) ) )

CSE152, Winter 2020

Epipolar Constraint: Calibrated Case

•  Two pinhole cameras with camera frames {1} and {2} •  The transformation between camera frames is given by •  Let each camera be calibrated with camera matrix M1 and M2. •  P projects to p in Camera 1 with 3D coordinates 1p, and P

projects to p’ in Camera 2 with 3D coordinates 2p’. •  What is the relation of the coordinates 1p and 2p’?

{1} {2}

21R and 1t2

f f’

10

CSE152, Winter 2020

Epipolar Constraint: Calibrated Case

Essential Matrix (Longuet-Higgins, 1981)

1p ⋅ 1t2 × ( 21R 2p ')⎡⎣ ⎤⎦= 0

The vectors , Op! "!

and are coplanar O ' p! "!!!

'OO '! "!!!

1pTE 2p ' = 0 with E = [(1t2 )×]21R

skew

CSE152, Winter 2020

Epipolar geometry example

11

CSE152, Winter 2020

How do you get 1p and 2p’ from image coordinates?

M = KΠwcT =

f s cx0 α f cy0 0 1

⎡

⎣

⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥

100

010

001

000

⎡

⎣

⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥

wcR cOw0T 1

⎡

⎣

⎢⎢

⎤

⎦

⎥⎥

K Intrinsic

Parameters

Π Projection

3 x 4

Extrinsic

Parameters

•  Mapping from point in homogenous world coordinates wP to homogenous pixel coordinates q

•  For the measured pixel coordinates q and q’ in cameras 1 and

2 with their own intrinsic parameterss, we have 1p = K1-1q

2p’ = K2-1 q’

q =M wP = KΠwcT wP = K c p where c p =Πw

cT wP

wcT

CSE152, Winter 2020

Two ways to estimate the Essential Matrix

1.  Calibration-based 2.  Eight-Point Algorithm

12

CSE152, Winter 2020

Calibration-Based Method

1.  From image of known calibration fixture, determine intrinsic parameters K1, K2 and extrinsic relation of two cameras: R1, R2, t1, t2

2.  Compute the relative position and orientation of the two cameras from R1, R2, t1, t2

3.  Compute the Essential Matrix E

21R and 1t2

CSE152, Winter 2020

The Eight-Point Algorithm (Longuet-Higgins, 1981)

uu ' uv ' u vu ' vv ' v u ' v ' 1⎡⎣⎢⎤⎦⎥

E11E12E13E21E22E23E31E32E33

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥

= 0

•  Set E33 to 1 •  Use 8 points (ui,vi), i=1..8

€

u,v,1[ ]E11 E12 E13E21 E22 E23E31 E32 E33

"

#

$ $ $

%

&

' ' '

u'v '1

"

#

$ $ $

%

&

' ' '

= 0

€

u1u'1 u1v'1 u1 v1u'1 v1v'1 v1 u'1 v'1u2u'2 u2v'2 u2 v2u'2 v2v'2 v2 u'2 v'2u3u'3 u3v'3 u3 v3u'3 v3v'3 v3 u'3 v'3u4u'4 u4v'4 u4 v4u'4 v4v'4 v4 u'4 v'4u5u'5 u5v'5 u5 v5u'5 v5v'5 v5 u'5 v'5u6u'6 u6v'6 u6 v6u'6 v6v'6 v6 u'6 v'6u7u'7 u7v'7 u7 v7u'7 v7v'7 v7 u'7 v'7u8u'8 u8v'8 u8 v8u'8 v8v'8 v8 u'8 v'8

"

#

$ $ $ $ $ $ $ $ $ $

%

&

' ' ' ' ' ' ' ' ' '

E11E12E13E21E22E23E31E32

"

#

$ $ $ $ $ $ $ $ $ $

%

&

' ' ' ' ' ' ' ' ' '

= −

11111111

"

#

$ $ $ $ $ $ $ $ $ $

%

&

' ' ' ' ' ' ' ' ' '

Solve E11 to E32 This are elements of the Essential Matrix

Input: 8 corresponding points in two images p=[ui,vi,1], p’=[ui’,vi’,1]

13

CSE152, Winter 2020

Epipolar geometry example

CSE152, Winter 2020

Example: converging cameras

courtesy of Andrew Zisserman

14

CSE152, Winter 2020

Example: motion parallel with image plane

(simple for stereo → rectification) courtesy of Andrew Zisserman

CSE152, Winter 2020

Example: forward motion

e

e’

courtesy of Andrew Zisserman

15

CSE152, Winter 2020

How do we use the Essential Matrix E?

•  Given a point in Image 1 with homogenous pixel coordinates q, convert it to a direction as 1p = (K1-1) q

•  Let a = 1pTE where a is 1 X 3 vector. •  Then apply the epipolar constraint, we have a 2p’ = 0 •  And converting direction to image coordinates with

2p’ = K2-1 q’, we have that (aK2-1) q’ =0

is the equation of the epipolar line in Image 2. •  Likewise, given a point q’ in Image 2, we can obtain a line equation in Image 1.

1pTE 2p ' = 0 with E = [(1t2 )×]21R

CSE152, Winter 2020

How do we use the Essential Matrix?

Given a point q in the left image, what is the equation of the corresponding epipolar line L in the right image?

q L

16

CSE152, Winter 2020

Computing the epipoles given E pTEp’ = 0 with E = [tx]R

•  The epipole e’ in the right image is the Eigenvector of E corresponding to the zero eigenvalue.

•  The epipole e in the left iamge is the Eigenvector of ET corresponding to the zero eigenvalue.

Why? 1.  det(E) = det([tx])det(R) = 0 because det([tx]=0 since [tx] is

skew symmetric. Consequently E has a zero Eigenvalue

CSE152, Winter 2020

Computing the epipoles from E

2.  Now, a vector is e’ is an Eigenvector of E iff Ee’ = λe’ where λ is the Eigenvalue.

3.  Since E has a zero Eigenvalue (λ=0), Ee’ =λe’= 0 4.  Consider a point e’ in the right image, where Ee’ = 0, the “line

equation” in the left image using the epipolar constraint is pTEe’ = 0 = pT0

But this isn’t a line. Every point p in the left image satisfies this equation. Therefore e’ is the epipole in the right image.

pTEp’ = 0 with E = [tx]R