View
6
Download
0
Category
Preview:
Citation preview
Lecture9:StereoandMVS
CSE152:ComputerVisionManmohanChandraker
Recap
epipolarlines
Disparity
(x1,y1) (x2,y1)
x2-x1=thedisparityofpixel(x1,y1)
Twoimagescapturedbyapurelyhorizontal translatingcamera(rectifiedstereopair)
Estimatingthedisparityisequivalenttoestimatingdepth.
Depth from disparity
f
x x’
baseline
z
C C’
X
f
Your basic stereo algorithm
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image• pick pixel with minimum match cost
Improvement: match windows
Popular matching scores
• SSD (Sum Squared Distance)
• NCC (Normalized Cross Correlation)
where
• What advantages might NCC have over SSD?
Plane sweep stereoRe-order (pixel and disparity) evaluation loops
for every pixel, for every disparityfor every disparity for every pixelcompute cost compute cost
Stereo matching framework1. For every disparity, compute raw matching
costs
Why use a robust function?• occlusions, other outliers
Can also use alternative match criteria.
Stereo matching framework2. Aggregate costs spatially
• Here, we are using a box filter(efficient moving averageimplementation)
• Can also use weighted average,or other filters
Stereo matching
I(x, y) J(x, y)
y = 141
E(x, y, d) the disparity space image (DSI)x
d
Stereo matching framework3. Choose winning disparity at each pixel
y = 141
E(x, y, d) the disparity space image (DSI)x
d
Stereo matching framework3. Choose winning disparity at each pixel
4. Interpolate to sub-pixel accuracy
d
E(d)
d*
width of a pixel
Choosingthestereobaseline
What’stheoptimalbaseline?
Large Baseline Small Baseline
width of a pixel
Choosingthestereobaseline
What’stheoptimalbaseline?– Toosmall:largedeptherror– Toolarge:difficultsearchproblem
Large Baseline Small Baseline
all of thesepoints projectto the same pair of pixels
Traditional Stereo MatchingAdvantages:
• gives detailed surface estimates• fast algorithms based on moving averages• sub-pixel disparity estimates and confidence
Limitations:• narrow baseline Þ noisy estimates• fails in textureless areas• gets confused near occlusion boundaries
Stereo as energy minimization
What defines a good stereo correspondence?1. Match quality
– Want each pixel to find a good match in the other image2. Smoothness
– If two pixels are adjacent, they should (usually) move about the same amount
Stereo as energy minimization
• Find disparity map d that minimizes an energy function
• Simple pixel or window matching
Match distance between windows I(x, y) and J(x + d(x,y), y)=
Stereo as energy minimizationBetter objective function
match cost smoothness cost
Want each pixel to find a good match in the other image
Adjacent pixels should (usually) move about the same
amount
Stereo as energy minimization
match cost:
smoothness cost:
4-connected neighborhood
8-connected neighborhood
: set of neighboring pixels
Stereo matching 20
Energy minimization1-D example: approximating splines
zx,y
dx,y
Options forsmoothness
RelaxationHow can we get the best solution?Differentiate energy function, set to 0
RelaxationIteratively improve a solution by locally
minimizing the energy: relax to solution
zx,y
dx,ydx-1,y dx+1,y
Graph cutsSolution technique for general 2D problem
Graph cutsTwo different kinds of moves:
Compute best possible match within integer disparity.
Smoothness cost
“Potts model”
L1 distance
How do we choose V?
Depth Map Results
Input image Sum Abs Diff
Graph cuts
CSE576,Spring2008 Stereomatching 27
Stereo evaluation
CSE576,Spring2008 Stereomatching 28
Stereo—best algorithms
Real-time stereo
Used for robot navigation (and other tasks)• Several software-based real-time stereo techniques have
been developed (most based on simple discrete search)
Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html
Multi-viewStereo
Figures by Carlos Hernandez
Input:calibratedimagesfromseveralviewpointsOutput:3Dobjectmodel
Stereo:anotherviewerror
depth
56 Flickr images taken by 8 photographers
State-of-the-artinbinocularstereo
[Zbontar andLeCun,JMLR2016]
Recommended