Multi-view Stereo via Multi-view Stereo via Volumetric Graph-cutsVolumetric Graph-cuts

George Vogiatzis, Philip H. S. Torr Roberto Cipolla

Shape From ImagesShape From Images

Dense Stereo Dense Stereo reconstruction problem:reconstruction problem:• Input

• Set of images of a scene I={I1,…,IK}

• Camera matrices P1,…,PK

• Output• Surface model

Shape parametrisationShape parametrisation

• Disparity-map parameterisation• MRF formulation – good optimisation

techniques exist (Graph-cuts, Loopy BP)

• MRF smoothness is viewpoint dependent

• Disparity is unique per pixel – only functions represented

Shape parametrisationShape parametrisation

• Volumetric parameterisation – e.g. Level-sets, Space carving etc.• Able to cope with non-functions• Convergence properties not well

understood• Memory intensive• For Space carving, no simple way to

impose surface smoothness

Solution ?Solution ?

• Cast volumetric methods in MRF framework

• Benefits:• General surfaces can be represented• Optimisation is tractable (MRF solvers)• Occlusions can be approximately

modelled• Smoothness is viewpoint independent

Graph cutsGraph cuts

40

30

23

12

13

5

540

24

1

50

3

4

2021

13

Graph cutsGraph cuts

40

30

23

12

13

5

540

24

1

50

3

4

2021

13

Graph cutsGraph cuts

40

30

23

12

13

5

540

24

1

50

3

4

2021

5+5+1+4+3=18

13

Volumetric Graph cuts for Volumetric Graph cuts for segmentationsegmentation

• Volume is discretized • A binary MRF is defined on the voxels • Regular grid (6 or 26 neighbourhood)• Voxels are labelled as OBJECT and

BACKGROUND• Labelling cost set by OBJECT /

BACKGROUND intensity statistics• Compatibility cost set by edge intensities

Volumetric Graph cuts for Volumetric Graph cuts for stereostereo• How to define ‘Inside’ and ‘Outside’

labels

• How to deal with occlusion

Volumetric Graph cutsVolumetric Graph cuts

Source

Sink

Min cut

FaceFace

Face - Visual Hull Face - Visual Hull

Face - SliceFace - Slice

Face - Slice with graphcutFace - Slice with graphcut

Face - ReconstructionFace - Reconstruction

Protrusion problemProtrusion problem

• ‘Balooning’ force• favouring bigger volumes

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

Protrusion problemProtrusion problem

• ‘Balooning’ force• favouring bigger volumes

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

Protrusion problemProtrusion problem

Protrusion problemProtrusion problem

GraphGraph

ResultsResults

• Model House

ResultsResults

• Model House – Visual Hull

ResultsResults

• Model House

ResultsResults• Stone carving

ResultsResults

• Haniwa

SummarySummary

• Novel formulation for multiview stereo

• Volumetric scene representation

• Computationally tractable global optimisation using Graph-cuts.

• Visual hull for occlusions and geometric constraint

BenefitsBenefits

1. General surfaces and objects can be fully represented and computed as a single surface.

2. The representation and smoothness constraint is image and viewpoint independent.

3. Multiple views of the scene can be used with occlusions approximately modelled.

4. Optimisation is computationally tractable, using existing max-flow algorithms.