View
226
Download
0
Tags:
Embed Size (px)
Citation preview
Last Time
Pinhole camera model, projection A taste of projective geometry Two view geometry:
Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix
Epipolar Lines
epipolar linesepipolar linesepipolar linesepipolar lines
BaselineBaselineOO O’O’
epipolar planeepipolar plane
' 0Tp Ep
Stereo Vision
Objective: 3D reconstruction Input: 2 (or more) images taken with calibrated
cameras Output: 3D structure of scene Steps:
Rectification Matching Depth estimation
Rectification
We will assume images have been rectified so that epipolar lines correspond to scan lines Image planes of cameras are parallel. Focal points are at same height. Focal lengths same.
Then, epipolar lines fall along the horizontal scan lines of the images
Any stereo pair can be rectified by rotating and scaling the two image planes (=homography) so that they become parallel to baseline
Rectification
Image Reprojection reproject image planes onto
common plane parallel to baseline Notice, only focal point of camera
really matters(Seitz)
Cyclopean Coordinates
Origin at midpoint between camera centers Axes parallel to those of the two (rectified) cameras
( / 2),
( / 2),
( ) ( ),
2 2
l l
r r
l r
l r l r
l r l r l r
f X b fYx y
Z Zf X b fY
x yZ Z
fbx x
Zb x x b y y fb
X Y Zx x x x x x
Disparity
The difference is called “disparity” d is inversely related to Z: greater sensitivity to
nearby points d is directly related to b: sensitivity to small
baseline
l r
fbZ
x x
l rd x x
Main Step: Correspondence Search What to match?
Objects?
More identifiable, but difficult to compute Pixels?
Easier to handle, but maybe ambiguous Edges? Collections of pixels (regions)?
Matching objects vs. Pixels
Left Right
scanline
Random Dot Stereogram
Using random dot pairs Julesz showed that recognition is not needed for stereo
Random Dot in Motion
Finding Matches
Under what conditions pixels can be matched? Ignoring specularities, we can assume that matching pixels
have the same brightness (constant brightness assumption)
Still, changes in gain and sensitivity may change the values of pixels
Common solution: Use larger windows Normalized correlation
Pros and cons: Small window: accurate match is more likely Large window: fewer candidates
We need a method to eliminate false matches
Window Size
W = 3 W = 20
Constraining the Search
Restrict search to epipolar lines (1D search) Use larger elements (larger windows, edges,
regions)
Problem: large elements may be distorted
Enforce smoothness
Problem: discontinuities at object boundaries
Enforce ordering
Problem: not always true
1D Search
SSD error
disparity
1D Search More efficient Fewer false matches
Ordering
Ordering
Correspondence as Optimization Most stereo algorithms attempt to minimize a
functional that usually consists of two terms:
where
- penalizes for quality of a match (unary)
- penalizes non smooth (or even non fronto-parallel) reconstructions (binary)
Many different optimization approaches were proposed
match data smoothnessE E E
smoothnessEdataE
Comparison of Stereo Algorithms
D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp. 7-42.
Ground truthScene
Scharstein and Szeliski
Results with window correlation
Window-based matching(best window size)
Ground truth
Graph Cuts
Ground truthGraph cuts
Stereo Algorithms
We’ll briefly review several algorithms: Dynamic programming Minimal cut/Max flow Space carving Graph cut optimization
?
1D Methods: Dynamic Programming Discretize the 3-D space Find the correct curve at every slice
(A slice = epipolar plane)
Dynamic programming
Find correspondences of each epipolar
line separately
Dynamic programming
Dynamic programming
How do we find the best curve? Assign weight of all edges
insertion
matchdeletion
Dynamic programming
How do we find the best curve? Assign weight of all edges Find shortest path
Dijkstra
insertion
matchdeletion
Results
Dynamic programming
Advantages Simple, efficient Globally optimal
Disadvantages Each slice computed independently
(smoothness is not enforced between slices) Problems due to discretization (tilted planes)
Min Cut/Max Flow
Min Cut/Max Flow
Min Cut/Max Flow
Min Cut/Max Flow
Min Cut/Max Flow
Objective: find the optimal cut using all the slices simultaneously.
Min Cut/Max Flow
Construct a graph: Every voxel (3-D point in space) is a node Every node is connected to its 6 neighbors
Min Cut/Max Flow
Weights on the edges: Data cost: change in pixel value
data
data
Min Cut/Max Flow
Weights on the edges: Data cost: change in pixel value Smoothness cost: change in depth
smooth
smooth
smooth
smooth
Min Cut/Max Flow
Weights on the edges: Data cost: change in pixel value Smoothness cost: change in depth
Min Cut/Max Flow
Add source and sink Find min cut
Source
…
…
Sink
∞
∞
Min Cut/Max Flow
Data penalty
Smoothness penalty
Results
Input Min cut Dynamic programming
Min Cut/Max Flow
Advantages All slices are optimized simultaneously Efficient
Disadvantages Extension to multi-camera is difficult Discretization
Multi-view stereo Every point in space
corresponds to a match in the images
Compute data term for each match
Space Carving
0.5 0.4 0.8 0.9 0.9 0.8 0.9 0.3 0.2
Space Carving
Multi-view stereo Every point in space
corresponds to a match in the images
Compute data term for each match (“photo-consistency”)
Space Carving
Dynamic data term (taking occlusion into account)
Order of sweep is important
Space Carving
Space Carving
Done for all slices simultaneously
Space Carving
Done for all slices simultaneously
Space Carving
Done for all slices simultaneously
Space Carving
Computes a bound on the object, the visual hull More camera views: better result
Space Carving: Results
Space Carving: Results
Space Carving
Advantages True multi-views stereo Handles occlusion
Disadvantages Limited to visual hull Lacks smoothness term Noise may introduce holes,
allowing for noise may thicken shape
Discretization
Graph Cut Optimization
Stereo is a minimization problem
Possible solution: local search (gradient descent) Problem: inefficient, local minima Instead, search larger areas at every iteration
match data smoothnessE E E
Graph Cut Optimization
Construct a graph to represent the problem: Nodes:
Pixels (in first image) k discrete depth values
Edges: From every pixel node to a
depth node (data term) Neighboring nodes (smoothness)
Assign weights corresponding to pixel intensities to get a global cost function
pixels
depths
…21 k
Graph Cut Optimization
Objective: Multiway cut Edges:
Every pixel remains connected to one depth node
Edges between neighboring nodes only if they are connected to same depth node
Nodes are assigned the depth that they are connected to
Multiway cut is NP-complete, solve iteratively
…
……21 k3
pixels
depths
Graph Cut Optimization
α-β swap Nodes labeled α or β, (i.e.,
connected to or )
can change their labeling to α or β
Edges between neighbors are updated according to the new labeling
Other edges are not changed Finding best swap = min cut!
α β
…
……21 k3
pixels
depths
Graph Cut Optimization
Example: 1-2 swap
…
…… k
…
…
… k1 2 1 23 3
Graph Cut Optimization
Example: 1-2 swap
…
…
… k1 2 3 … k321
Connect the nodes labeled 1 or 2 to both
labels
Graph Cut Optimization
Example: 1-2 swap… k3
2
1 … k3
2
1
Mark 1 as source and 2 as sink Find minimal cut
Graph Cut Optimization
Example: 1-2 swap
… k3
2
1
… k1 2 3
Erase edges that were on the cut
Result: a new labeling of the 1,2 nodes
Graph Cut Optimization
Start with an arbitrary labeling For every pair {α, β} є {1,…,k}
Find the best α-β swap (minimizing the function) Update the graph (add and erase edges)
Quit when no pair improves the cost function Induce pixel labels
Graph Cuts: Results
…21 k3
Advantages State of the art results Efficient Bound on approximation quality Same technique can be applied to other
problems (e.g., image restoration)Disadvantages Discretization Occlusion Still room for improvement
Graph Cut Optimization
Summary
Stereo vision: shape reconstruction from two or more images
Steps: Rectification Correspondence search Depth estimation
Algorithms: Dynamic programming Min cut/max flow Space carving Graph cuts