Midterm Shape-from-X (i.e., Reconstruction) · CSE 152 Lecture 13 CSE152, Spr 04 Intro Computer Vision ... Turn in hardcopy to my office 3101 AP&M • No Discussion section this week

1

CSE152, Spr 04 Intro Computer Vision

Stereo Vision II

Introduction to Computer VisionCSE 152

Lecture 13


Announcements

• Assignment 3: Due today.– Extended to 5:00PM, sharp. Turn in hardcopy to

my office 3101 AP&M• No Discussion section this week.• Guest lecturer, Dr. Jeff Ho, next week.• Today

– Midterm summary– Shape from X– Stereo Vision I


Midterm• High: 87• Low: 31• Mean: 57• Median: 55

Problem regrades: Must be requested today in class and in writing.


Shape-from-X(i.e., Reconstruction)

• Methods for estimating 3-D shape from image data. X can be one (or more) of many cues.– Stereo (two or more views, known viewpoints)– Motion (moving camera or object)– Shading– Changing lighting (Photometric Stereo)– Texture variation– Focus/blur– Prior knowledge/context– structured light/lasers


Example: Helmholtz StereoDepth + Normals + BRDF


Stereo

2


Binocular Stereopsis: MarsGiven two images of a scene where relative locations of cameras are known, estimate depth of all common scene points.

Two images of Mars


An Application: Mobile Robot Navigation

The Stanford Cart,H. Moravec, 1979.

The INRIA Mobile Robot, 1990.

Courtesy O. Faugeras and H. Moravec.


Commercial Stereo Heads

Trinocular stereoTrinocular stereo Binocular stereoBinocular stereo


Stereo can work well


Need for correspondence

Truco Fig. 7.5


Triangulation

Nalwa Fig. 7.2

3


Stereo Vision Outline• Offline: Calibrate cameras & determine

“epipolar geometry”• Online

1. Acquire stereo images2. Rectify images to convenient epipolar geometry3. Establish correspondence 4. Estimate depthA

B

C

D


BINOCULAR STEREO SYSTEMEstimating Depth

Z

X(0,0) (d,0)

Z=fXL XR

DISPARITY(XL - XR)

Z = (f/XL) XZ= (f/XR) (X-d)

(f/XL) X = (f/XR) (X-d)X = (XLd) / (XL - XR)

Z = d f(XL - XR)

X = d XL(XL - XR)

(Adapted from Hager)


Reconstruction: General 3-D case

• Linear Method: find P such that

• Non-Linear Method: find Q minimizingCSE152, Spr 04 Intro Computer Vision

Two Approaches• A) From each image, process “monocular”

image to obtain cues.B) Establish correspondence between

cues.• Directly compare image regions between

the two images.


Human Stereopsis: Binocular Fusion

How are the correspondences established?

Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one??• There is anecdotal evidence for the latter (camouflage).

• Random dot stereograms provide an objective answer BP!CSE152, Spr 04 Intro Computer Vision

Random Dot Stereograms

4


Random Dot Stereograms


A Cooperative Model (Marr and Poggio, 1976)


Epipolar Constraint

• Potential matches for p have to lie on the corresponding epipolar line l’.

• Potential matches for p’ have to lie on the corresponding epipolar line l.


Epipolar Geometry

• Epipolar Plane

• Epipoles

• Epipolar Lines

• Baseline


Family of epipolar Planes(standard approach)


JEFF, HERE’s WHERE I ENDED• Jeff, I covered up to this point.• I suggest that you

– review the pictorial part of the epipolar geometry, • discuss the essentnial matrix, showing how it can

be computed from R&T, and that this can come from calibration.

• Discuss rectification in qualitative way, showing that result is epipolar lines become parallel lines.

• Then discuss matching, mostly SSD & SAD metric.

• The mention issue with half occluded regions.• Perhaps mention some challenges, dynamic

programming, etc.

5


Family of epipolar Planes(standard approach)


Epipolar Constraint: Calibrated Case

Essential Matrix(Longuet-Higgins, 1981)


Properties of the Essential Matrix

• E p’ is the epipolar line associated with p’.

• ETp is the epipolar line associated with p.

• E e’=0 and ETe=0.

• E is singular.

• E has two equal non-zero singular values(Huang and Faugeras, 1989).

T

T


Calibration

Determine intrinsic parameters and extrinsic relation of two cameras


The Eight-Point Algorithm (Longuet-Higgins, 1981)

|F | =1.

Minimize:

under the constraint2

Set F33 to 1


Epipolar geometry example

6


Example: converging cameras

courtesy of Andrew Zisserman CSE152, Spr 04 Intro Computer Vision

Example: motion parallel with image plane

(simple for stereo → rectification)courtesy of Andrew Zisserman


Example: forward motion

e

e’

courtesy of Andrew Zisserman CSE152, Spr 04 Intro Computer Vision

RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.


Rectification

All epipolar lines are parallel in the rectified image plane.CSE152, Spr 04 Intro Computer Vision

Image pair rectification

simplify stereo matching by warping the images

Apply projective transformation so that epipolar linescorrespond to horizontal scanlines

e

e

map epipole e to (1,0,0)try to minimize image distortion

He001

=

Note that rectified images usually not rectangular

7



Input Images



Rectified ImagesSee Section 7.3.7 for specific method


Features on same epipolar line

Truco Fig. 7.5


Mobi: Stereo-based navigation


Epipolar correspondence

This version is feature-based: detect edges in 1-D signal, and use dynanic progrmaming toe find correspondences that minimize an error function.


A challenge: Multiple Interpretations

Each feature on left epipolar line match oneand only one feature on right epipolar line.

8


Multiple Interpretations









Dense Correspondence: A Photometric constraint

• Same world point has same intensity in both images (Constant Brightness Constraint)– Lambertian fronto-parallel– Issues:

• Noise• Specularity• Foreshortening


Using epipolar & constant Brightness constraints for stereo matching

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost• This will never work, so:

Improvement: match windows

(Seitz)CSE152, Spr 04 Intro Computer Vision

Comparing Windows: ==??

ff gg

MostMostpopularpopular

(Camps)

For each window, match to closest window on epipolar line in other image.

9


Match Metric Summary

( )( ) ( )( )

( )( ) ( )( )∑ ∑

∑

−+⋅−

−+⋅−

vu vu

vu

IvduIIvuI

IvduIIvuI

, ,

222

211

,2211

,,

,,

( ) ( )( )∑ +−vu

vduIvuI,

221 ,,

( )( )( )( )

( )( )( )( )

∑∑∑

−+

−+−

−

−

vu

vuvu

IvduI

IvduI

IvuI

IvuI,

2

,

222

22

,

211

11

,

,

,

,

( ) ( )∑ +−vu

vduIvuI,

21 ,,

( ) ( )( )∑ +−vu

vduIvuI,

'2

'1 ,,

( ) ( ) ( )∑ <=nm

kkk vuInmIvuI,

' ,,,

( ) ( )( )∑ +vu

vduIvuIHAMMING,

'2

'1 ,,,

( ) ( ) ( )( )vuInmIBITSTRINGvuI kknmk ,,, ,' <=

MATCH METRIC DEFINITION

Normalized Cross-Correlation (NCC)

Sum of Squared Differences (SSD)

Normalized SSD

Sum of Absolute Differences (SAD)

Zero Mean SAD

Rank

Census

These two are actually

the same

( ) ( )∑ −+−−vu

IvduIIvuI,

_

22

_

11 ),(),(


Correspondence Search Algorithm (simple version for Cross Correlation)

For i = 1:nrowsfor j=1:ncols

best(i,j) = -1for k = mindisparity:maxdisparity

c = CC(I1(i,j),I2(i,j+k),winsize)if (c > best(i,j))

best(i,j) = cdisparities(i,j) = k

endend

endend

O(nrows * ncols * disparities * winx * winy)

I1 I2

uv

d

I1 I2

uv

d


Stereo results

Ground truthScene

– Data from University of Tsukuba


Results with window correlation

Window-based matching(best window size)

Ground truth

(Seitz)


Results with better method

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth


Window size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi, A Stereo Matching

Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

• Effect of window size

(Seitz)

10


Stereo ConstraintsCONSTRAINT BRIEF DESCRIPTION

1-D Epipolar Search Arbitrary images of the same scene may be rectified based on epipolar geometry such that stereo matches lie along one-dimensional scanlines. This reduces the computational complexity and also reduces the likelihood of false matches.

Monotonic Ordering Points along an epipolar scanline appear in the same order in both stereo images, assuming that all objects in the scene are approximately the same distance from the cameras.

Image Brightness Constancy

Assuming Lambertian surfaces, the brightness of corresponding points in stereo images are the same.

Match Uniqueness For every point in one stereo image, there is at most one corresponding point in the other image.

Disparity Continuity Disparities vary smoothly (i.e. disparity gradient is small) over most of the image. This assumption is violated at object boundaries.

Disparity Limit The search space may be reduced significantly by limiting the disparity range, reducing both computational complexity and the likelihood of false matches.

Fronto-Parallel Surfaces

The implicit assumption made by area-based matching is that objects have fronto-parallel surfaces (i.e. depth is constant within the region of local support). This assumption is violated by sloping and creased surfaces.

Feature Similarity Corresponding features must be similar (e.g. edges must have roughly the same length and orientation).

Structural Grouping Corresponding feature groupings and their connectivity must be consistent.

(From G. Hager) CSE152, Spr 04 Intro Computer Vision

Problem of Occlusion


Stereo Matching using Dynamic Programming

Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.

(Ohta and Kanade, 1985)


Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar• ordering• uniqueness• disparity limit• disparity gradient limit

Trade-off• Matching cost (data)• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)(From Pollefeys)


Some Challenges & Problems• Photometric issues:

– specularities– strongly non-Lambertian BRDF’s

• Surface structure– lack of texture– repeating texture within horopter bracket

• Geometric ambiguities– as surfaces turn away, difficult to get accurate reconstruction

(affine approximate can help)– at the occluding contour, likelihood of good match but incorrect

reconstruction


Variations on Binocular Stereo1. Trinocular Stereopsis2. Helmholtz Reciprocity Stereopsis

11


Trinocular Epipolar Constraints

These constraints are not independent!


More on stereo …

The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/

Recommended reading

D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR-2001-81, November 2001.

Myron Z. Brown, Darius Burschka, and Gregory D. Hager. Advances in Computational Stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8):993-1008, 2003.

Documents

Midterm Shape-from-X (i.e., Reconstruction) · CSE 152 Lecture 13 CSE152, Spr 04 Intro Computer Vision ... Turn in hardcopy to my office 3101 AP&M • No Discussion section this week