View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Samuel W. Hasinoff Sing Bing Kang Richard Szeliski
Interactive Visual Media GroupMicrosoft Research{sbkang,szeliski}@microsoft.com
Dept. of Computer ScienceUniversity of [email protected]
Boundary Matting for View Synthesis
2nd Workshop on Image and Video Registration, July 2, 2004
MotivationSuperior view synthesis & 3D editing from N-view stereo
Key approach: occlusion boundaries as 3D curves
• More suitable for view synthesis• Boundaries estimated to sub-pixel
Two major limitations – even with perfect stereo!• Resampling blur• Boundary artifacts
B2B3
Matting problem: Unmix the foreground & background
Matting from Stereo
BFC )1(
Triangulation matting (Smith & Blinn, 1996)
• multiple backgrounds• fixed viewpoint & object
F
B1
Extension to stereo• Lambertian assumption
F
B3B1 B2
underdetermined
Occlusion Boundaries in 3D Model boundaries as 3D splines (currently linear) Assumptions
boundaries are relatively sharp relatively large-scale objects no internal transparency
view 1 view 3view 2 (reference)
3D world
Geometric View of Alpha
alpha partial pixel coverage on F side
simulate blurring by convolving with 2D Gaussian
otherwise,0
0)(,1)(
xdx
),0()(),( Gxx
j
j x)(
alpha depends only on projected 3D curve, x
integration over each pixel
F B
pixel j
Related Work Natural image matting [Chuang et al., 2001]
based on color statistics
Intelligent scissors [Mortenson, 2000]
geometric view of alpha
- single image- user-assisted
Related Work Bayesian Layer estimation [Wexler and Fitzgibbon, 2002]
matting from multiple images using triangulation + priors
- requires very high-quality stereo- alpha calculated at pixel level, only for reference - not suitable for view synthesis
Boundary Matting Algorithm
3D world
view 1 view 3view 2 (reference)
find occlusion boundary in reference view backproject to 3D using stereo depth project to other views initial guess for Bi and F optimize matting
optimize
Initial Boundaries From Stereo Find depth discontinuities Greedily segment longest four-connected curves
Spline control points evenly spaced along curve
Tweak - snap to strongest nearby edge
Background Estimation
F
B1 B2
Use stereo to grab corresponding background-depth pixels from nearby views (if possible)
Color consistency check to avoid mixed pixels
B3
occluded
Foreground Estimation
Invert matting equation, given 3D curve and B
Aggregate F estimates over all views
viewsviews
ii
iii FF
1
2
1
2 )(ˆˆ
BCF )1()(ˆ
BFC )1(
Optimization
Objective: Minimize inconsistency with matting
over curve parameters, x, and foreground colors, F
Pixels with unknown B not included Non-linear least squares, using forward differencing
for Jacobian
views pixels
i j
jijijjiji BαFαCO1 1
2))(1()(),( xxFx
Additional Penalty Terms Favor control points at strong edges
define potential field around each edgel
Discourage large motions (>2 pixels) helps avoid degenerate curves
Naïve object insertion (no matting)
Object insertion with Boundary Matting
Naïve object insertion (no matting)
Object insertion with Boundary Matting
Naïve object insertion (no matting)
Object insertion with Boundary Matting
boundaries calculated with subpixel accuracy
Samsung commercial sequence
Naïve object insertion (no matting)
Object insertion with Boundary Matting
Boundary Matting Naïve method
Boundary Matting Naïve method
boundary mattingboundary matting (sigma = 13)boundary matting (sigma = 26)compositebackgroundno matting
Synthetic Noise
Concluding Remarks
Boundary Matting better view synthesis refines stereo at occlusion boundaries subpixel boundary estimation
Future work incorporate color statistics extend to dynamic setting