Carved Visual Hulls for Image-Based Modeling

Carved Visual Hulls for Image-Based ModelingYasutaka Furukawa, Jean Ponce

presented byPhongsathorn Eakamongul

Department of Computer ScienceAsian Institute of Technology

2009, July 27

Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 1 / 49

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

5 Global Optimization

6 Local Refinement

7 Result

8 Comparison

9 Limitation & Future work

Reference

silhouette image from http://www.flickr.com/photos/alexfrance/3693058077/

space carving, shadow curving, voxel coloring from Silvio Savarese’s slide.

http://www.cs.cornell.edu/courses/cs664/2005fa/Lectures/lecture15.pdf

graph cut from ECCV 2006 tutorial

Multi-view shape reconstruction, lecture from Dana Cobzas, PIMS Postdoc

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Introduction

image-based Modeling

optimization

accuracy 1/200 ( 1 mm for 20 cm wide object ) from a set of low resolution(640x480px2) images∗

∗ From a comparison and evaluation of multi-view stereo reconstruction algorithms by ( Seitz et al. 2006 ), comparision of different method also available at

http://vision.middlebury.edu/mview/

choose a surface representation

define a photo-consistency function ( discrepancy between different projectionsof their surface points )

solve the following minimization

Representation

Volumetric representation

shape from silhouette ( Visual Hull )

space carving

voxel coloring

others i.e. Voxel-based, Image ray based, Axis-aligned, etc.

shape from silhouette ( Visual Hull )image object’s contour

Why contour ?

No texture

No shading

shape from silhouettevisual cones intersection

Carve all voxels outside the cone

Not photo-consistent (only to binary images)

Space carving algorithm

Initialize to a volume V containing the true scene

Choose a voxel on the current surface

Project to visible input images

Carve away voxels if not photo-consistent with images

Repeat until convergence

Voxel Coloring

Assign colors (RGBA : color + opacity) to voxels consistent with the input images.

Optimization

Graph cut

find the minimum cut , by compute the maximum flow , and look for the cut(s) thatseparate origin and destination by cutting through bottlenecks of the network.

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

shape from silhouette

initialize deformation of a surface mesh under photoconsistency constriantsoutput : rims that are used in graph cuts

global optimization process : use graph cuts with photoconsistency constraints +geometric contraints ( rims )

local refinement : enforce geometric contraints + photoconsistency constraints

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Identify Rims

Identify Rimscone strips

( Lazebnik et al 2007 ) represent visual hull in terms of polyhedral cone stripsΓ : rim

Identify Rimscone strips

Identify Rimstriangulated mesh model

( Lazebnik et al 2007 ) algorithm to obtain triangulated mesh model

In practices, measurement errors result to multiple horizontal neighbour

Identify RimsImage Discrepancy

Since rim segments are only part that touch surface of object, they can be foundas strip curves that minimize some measure of image discrepancy

Image Discrepancy Score ( Faugeras and Keriven 1998 )

f (p) = 2τ(τ−1)

Pτi=1

Pτj=i+1 1− exp(− (1−NCC(hi ,hj ))2

NCC(hi , hj ) : normalizated cross correlation between hi and hj

Identify Rims

image discrepancy is smallest values, so find shortest path ; path length = imagediscrepancy function

find shortest path by dynamic programming

Identify Rimsfind shortest path by dynamic programming

rim can be discontinous due to T-junction

First, Assume rim discontinuities occur only at right or left end points of eachconnected strip component

change from undirected graph to direct graph and apply dynamic programming

result :

Identify Rimsremove false rim segmentation

assumption break at complicated structure i.e. the fold of human clothremove false rim segmentation

Among all vertics identified as rim points, filter out false-positivesvertex v is detected as false positive if

either 4Rl < g(v) or Rl < g(v) [vertical size is too large] andf∗(v) < η [vertical size is not small enough] and [average NCC wrose than η]

average NCC score ( Faugeras and Keriven 1998 )

f ∗(p) = 2τ(τ−1)

Pτi=1

Pτj=i+1 NCC(hi , hj )

Identify Rims

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Global Optimization

use rim Γ to split surface Ω into Gi (i = 1, ..., k)

iteratively deform Gi inwards to generate multiple layers of 3D graph Ji andassociate photoconsistency weights to the edge of this graph

use graph cuts to carve surface

Global OptimizationMove Vertices down

At each iteration, move every vertex v along its surface normal N(v) and applysmoothing

v ← v − ελ

(ζ1f (v) + ζ2)N(v) + s(v)

notice that surface shrinks faster where image discrepancy function is largerConstant they use in all their experimentζ1 = 100, ζ2 = 0.1, β1 = 0.4, β2 = 0.3, λ = 20, ρ = 40, ε = 0.5 ∗ AverageLengthInGi

Global OptimizationBuilding a graph and Apply Graph Cuts

build vertical edgebuild horizontal edgeassign vertical and horizontal edge weight

weight of edge (vi , vj )

wij =α(f (vi )+f (vj )(δi +δj ))

d(vi ,vj )

f (vi ) : photoconsistency function; d(vi , vj ) : length of edge; δi : sparity of verticesaround vi , α = 1.0 for horizontal, 6.0 for vertical∞ edge weight for edge connected to source and sinkapply graph cuts

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Local Refinement

Local Refinementlocal minimum

increase resolution of mesh ( Hoppe et al 1993 ) until image projections of edgesbecome approximiately 2 px in length

rim consistency ( Hernandex Esteban and Schmitt 2004 )

r(vk ) = ¯v∗j rjexp(− ¯vk rj− ¯v∗k rj )

2/2σ22P

v′k∈Vjexp(−( ¯vk′ rj− ¯v∗j rj )2/2σ22)

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Result7 data sets

Result

ResultRim Identification Result

Filtering ratio : how many % of identified rim points has been filtered out as outliers ( for eachcontour )Sizes of components : show 3 largest connected components inside identified rim-segments

From table, visual hull boundary is mostly covered by a single large connected component exceptfor Twin data set, which has many input images, and hence, many rim curves.

ResultRunning Time (with 3.4 GHz Pentium 4)

bottleneck of computation is global optimization and local refinement step ( takes about2 hr )

ResultTesting algorithm w/o graph cuts phase ( use only local method )

local minimum problem

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Comparisons

( This paper ) rim constriants( Voiazis et al 2005 ) add inflationary ballooning term to enery function in graph cuts toprevent over-carving

Comparisons

( Hernandex Exteban and Schmitt 2004 ) In local refinement, use gradient flow insteadof direct derivatives 5f (v)Some other differences, i.e. local iterative deformation, that have problem avoidinglocal minima.

mistake in rim-identification stepsoutperforms in reconstructingconcave structure

Comparisons

ComparisonsMulti-view stereo evaluation ( http://vision.middlebury. edu/mview/ )

Temple data set

Accurracy : distance d that bring 90% of result within ground-truth surfaceCompleteness : % ground-truth surface that lies within 1.25 mm of the result

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

6 Local Refinement

7 Result

8 Comparison

Limitation & Future work

Since, cannot handle concavities too deep to be carved by the graph cuts. i.e. eyesockets of skullsTo overcomes this, combine their works with sparse wide-baseline stereo frominterest point (e.g. Schaffalitzky and Zisserman 2001) in order to incorporatestronger geometric constraints in the carving and local refinement stages

handle non-Lambertain surfaces ( Soatto et al 2003 )

simutaneous camera calibration where both camera parameters and surfaceshape are refined simultaneously using bundle adjustment ( Uffenkamp 1993 )

Carved Visual Hulls for Image-Based Modeling

Education

Projective Visual Hulls · hull by taking advantage of its intrinsically projective structure determined by special kinds of contacts between visual rays and the object’s surface

BUREAU OF ANIMAL INDUSTRY · the grains, rice hulls, cane sugar, bagasse, dried beet pulp, oat hulls, barley hulls, clipped oat by-products, sorghum plant by-products and flax plant

Hydrodynamic Evaluation of Planing Hulls

All Procedures Carved out of APGs (individual codes … Procedures Carved out of APGs ... Procedure carve out : ... All Procedures Carved out of APGs (individual codes carved out &

Hydrodynamics of single-deadrise hulls and their … · Asymmetric planing hulls are often ... deadrise hulls were conducted by Savitsky ... Empirical correlations for hydrodynamic

Sailboat Hulls

Visual Hulls for Volume Estimation: A Fast Marching Cubes

Introduction & Convex Hulls - ITI Algorithmik

Image-Based Visual Hulls - Harvard University

Rational Design of FPSO Hulls

Institute Time Clauses (Hulls) (ITCH) vs Nordic Plan€¦ · Institute Time Clauses (Hulls) (ITCH) vs Nordic Plan This is an overview of the Institute Time Clauses Hulls 1983 (ITCH)

Carved Visual Hulls for Image-Based Modelingcs.ait.ac.th/~mdailey/cvreadings/Furukawa-VisualHulls.pdf · 2009. 7. 27. · Département d’Informatique, Ecole Normale Supérieure,

CONVEX HULLS IN COARSE MEDIAN SPACEShomepages.warwick.ac.uk/~masgak/papers/hulls-cms.pdfThe aim of this paper is to describe the geometry of coarse convex hulls in coarse median spaces

1 Image-Based Visual Hulls Paper by Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J. Gortler and Leonard McMillan [wojciech/vh/]

All Procedures Carved out of APGs (individual codes carved ... · APG Description All Procedures Carved out of APGs (individual codes carved out & codes that group to carved out APGs)

Incorporating Dynamic Real Objects into Immersive … · Web viewAt SIGGRAPH 2000, Matusik, et al., presented an image-based visual hull algorithm, “Image Based Visual Hulls”

Parametric Generation of Yacht Hulls

Eﬃcient characteristic hulls - geosensor.net

Image-Based Visual Hulls

Carved Space