Carved Visual Hulls for Image-Based Modeling

Carved Visual Hulls for Image-Based ModelingYasutaka Furukawa, Jean Ponce

presented byPhongsathorn Eakamongul

Department of Computer ScienceAsian Institute of Technology

2009, July 27

Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 1 / 49

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

5 Global Optimization

6 Local Refinement

7 Result

8 Comparison

9 Limitation & Future work


Reference

silhouette image from http://www.flickr.com/photos/alexfrance/3693058077/

space carving, shadow curving, voxel coloring from Silvio Savarese’s slide.

http://www.cs.cornell.edu/courses/cs664/2005fa/Lectures/lecture15.pdf

graph cut from ECCV 2006 tutorial

Multi-view shape reconstruction, lecture from Dana Cobzas, PIMS Postdoc


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Introduction

image-based Modeling

optimization



accuracy 1/200 ( 1 mm for 20 cm wide object ) from a set of low resolution(640x480px2) images∗

∗ From a comparison and evaluation of multi-view stereo reconstruction algorithms by ( Seitz et al. 2006 ), comparision of different method also available at

http://vision.middlebury.edu/mview/


http://vision.middlebury.edu/mview/


choose a surface representation

define a photo-consistency function ( discrepancy between different projectionsof their surface points )

solve the following minimization


Representation


Volumetric representation

shape from silhouette ( Visual Hull )

space carving

voxel coloring

others i.e. Voxel-based, Image ray based, Axis-aligned, etc.


shape from silhouette ( Visual Hull )image object’s contour

Why contour ?

No texture

No shading


shape from silhouettevisual cones intersection

Carve all voxels outside the cone

Not photo-consistent (only to binary images)


Space carving algorithm

Initialize to a volume V containing the true scene

Choose a voxel on the current surface

Project to visible input images

Carve away voxels if not photo-consistent with images

Repeat until convergence


Voxel Coloring

Assign colors (RGBA : color + opacity) to voxels consistent with the input images.


Optimization


Graph cut

find the minimum cut , by compute the maximum flow , and look for the cut(s) thatseparate origin and destination by cutting through bottlenecks of the network.


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Steps

shape from silhouette

initialize deformation of a surface mesh under photoconsistency constriantsoutput : rims that are used in graph cuts


Steps

global optimization process : use graph cuts with photoconsistency constraints +geometric contraints ( rims )

local refinement : enforce geometric contraints + photoconsistency constraints


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Identify Rims


Identify Rimscone strips

( Lazebnik et al 2007 ) represent visual hull in terms of polyhedral cone stripsΓ : rim


Identify Rimscone strips


Identify Rimstriangulated mesh model

( Lazebnik et al 2007 ) algorithm to obtain triangulated mesh model

In practices, measurement errors result to multiple horizontal neighbour


Identify RimsImage Discrepancy

Since rim segments are only part that touch surface of object, they can be foundas strip curves that minimize some measure of image discrepancy

Image Discrepancy Score ( Faugeras and Keriven 1998 )

f (p) = 2τ(τ−1)

Pτi=1

Pτj=i+1 1− exp(− (1−NCC(hi ,hj ))2

2σ21

)

NCC(hi , hj ) : normalizated cross correlation between hi and hj


Identify Rims

image discrepancy is smallest values, so find shortest path ; path length = imagediscrepancy function

find shortest path by dynamic programming


Identify Rimsfind shortest path by dynamic programming

rim can be discontinous due to T-junction

First, Assume rim discontinuities occur only at right or left end points of eachconnected strip component

change from undirected graph to direct graph and apply dynamic programming

result :


Identify Rimsremove false rim segmentation

assumption break at complicated structure i.e. the fold of human clothremove false rim segmentation

Among all vertics identified as rim points, filter out false-positivesvertex v is detected as false positive if

either 4Rl < g(v) or Rl < g(v) [vertical size is too large] andf∗(v) < η [vertical size is not small enough] and [average NCC wrose than η]

average NCC score ( Faugeras and Keriven 1998 )

f ∗(p) = 2τ(τ−1)

Pτi=1

Pτj=i+1 NCC(hi , hj )


Identify Rims


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Global Optimization


Global Optimization

use rim Γ to split surface Ω into Gi (i = 1, ..., k)

iteratively deform Gi inwards to generate multiple layers of 3D graph Ji andassociate photoconsistency weights to the edge of this graph

use graph cuts to carve surface


Global OptimizationMove Vertices down

At each iteration, move every vertex v along its surface normal N(v) and applysmoothing

v ← v − ελ

(ζ1f (v) + ζ2)N(v) + s(v)

notice that surface shrinks faster where image discrepancy function is largerConstant they use in all their experimentζ1 = 100, ζ2 = 0.1, β1 = 0.4, β2 = 0.3, λ = 20, ρ = 40, ε = 0.5 ∗ AverageLengthInGi


Global OptimizationBuilding a graph and Apply Graph Cuts

build vertical edgebuild horizontal edgeassign vertical and horizontal edge weight

weight of edge (vi , vj )

wij =α(f (vi )+f (vj )(δi +δj ))

d(vi ,vj )

f (vi ) : photoconsistency function; d(vi , vj ) : length of edge; δi : sparity of verticesaround vi , α = 1.0 for horizontal, 6.0 for vertical∞ edge weight for edge connected to source and sinkapply graph cuts


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Local Refinement


Local Refinementlocal minimum

increase resolution of mesh ( Hoppe et al 1993 ) until image projections of edgesbecome approximiately 2 px in length

rim consistency ( Hernandex Esteban and Schmitt 2004 )

r(vk ) = ¯v∗j rjexp(− ¯vk rj− ¯v∗k rj )

2/2σ22P

v′k∈Vjexp(−( ¯vk′ rj− ¯v∗j rj )2/2σ22)


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Result7 data sets


Result


ResultRim Identification Result

Filtering ratio : how many % of identified rim points has been filtered out as outliers ( for eachcontour )Sizes of components : show 3 largest connected components inside identified rim-segments

From table, visual hull boundary is mostly covered by a single large connected component exceptfor Twin data set, which has many input images, and hence, many rim curves.


ResultRunning Time (with 3.4 GHz Pentium 4)

bottleneck of computation is global optimization and local refinement step ( takes about2 hr )


ResultTesting algorithm w/o graph cuts phase ( use only local method )

local minimum problem


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Comparisons

( This paper ) rim constriants( Voiazis et al 2005 ) add inflationary ballooning term to enery function in graph cuts toprevent over-carving


Comparisons

( Hernandex Exteban and Schmitt 2004 ) In local refinement, use gradient flow insteadof direct derivatives 5f (v)Some other differences, i.e. local iterative deformation, that have problem avoidinglocal minima.

mistake in rim-identification stepsoutperforms in reconstructingconcave structure


Comparisons


ComparisonsMulti-view stereo evaluation ( http://vision.middlebury. edu/mview/ )

Temple data set

Accurracy : distance d that bring 90% of result within ground-truth surfaceCompleteness : % ground-truth surface that lies within 1.25 mm of the result


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Refinement

7 Result

8 Comparison



Limitation & Future work

Since, cannot handle concavities too deep to be carved by the graph cuts. i.e. eyesockets of skullsTo overcomes this, combine their works with sparse wide-baseline stereo frominterest point (e.g. Schaffalitzky and Zisserman 2001) in order to incorporatestronger geometric constraints in the carving and local refinement stages

handle non-Lambertain surfaces ( Soatto et al 2003 )

simutaneous camera calibration where both camera parameters and surfaceshape are refined simultaneously using bundle adjustment ( Uffenkamp 1993 )


Education

Carved Visual Hulls for Image-Based Modeling