Using Strong Shape Priors for Multiview Reconstruction

Yunda Sun Pushmeet Kohli

Mathieu Bray Philip HS Torr

Department of Computing

Oxford Brookes University

Objective

+

[Images Courtesy: M. Black, L. Sigal]

Parametric Model

Images

Silhouettes

Pose

Estimate

Reconstruction

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Multiview Reconstruction

Need for Shape Priors

Multiview Reconstruction No Priors

• Silhouette Intersection• Space Carving

Weak Priors• Surface smoothness

– Snow et al. CVPR ’00

• Photo consistency and smoothness

– Kolmogorov and Zabih [ECCV ’02]

– Vogiatzis et al. [CVPR ’05] [Image Courtesy: Vogiatzis et al.]

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Shape-Priors for Segmentation

OBJ-CUT [Kumar et al., CVPR ’05]• Integrate Shape Priors in a MRF

POSE-CUT [Bray et al., ECCV ’06] • Efficient Inference of Model Parameters

Parametric Object Models as Strong Priors

Layered Pictorial Structures

Articulated Models

Deformable Models

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation and Reconstruction Results

Object-Specific MRF

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

x : Voxel label θ : Model Shape

Object-Specific MRF

Shape Prior

x : Voxel label θ : Model Shape

: shortest distance of voxel i from the rendered model

Object-Specific MRF

Smoothness Prior

x : Voxel label θ : Model Shape

Potts Model

Object-Specific MRF

Unary Likelihood

x : Voxel label θ : Model Shape : Visual Hull

For a soft constraint we use a large constant K instead of infinity

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

Can be solved using Graph cuts

[Kolmogorov and Zabih, ECCV02 ]

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

How to find the optimal Pose?

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Inference of Pose Parameters

Rotation and Translation of Torso in X axes

Rotation of left shoulder in X and Z axes

Inference of Pose Parameters

Minimize F(ө) using Powell Minimization

Let F(ө) =

Computational Problem:

Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT..

Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Experiments

Deformable Models

Articulated Models• Reconstruction Results• Human Pose Estimation

Deformable Models

Four Cameras 1.5 x 105 voxels DOF of Model: 5

Visual Hull

Our Reconstruction

Shape Model

Articulated Models

Articulated Models

Four Cameras 106 voxels DOF of Model: 26

Shape Model

Camera Setup

Articulated Models

500 function evaluations of F(θ) required Time per evaluation: 0.15 sec Total time: 75 sec

Let F(ө) =

Articulated Models

Visual Hull

Our Reconstruction

Pose Estimation Results

Visual Hull

Reconstruction

Pose Estimate

Pose Estimation Results

Quantitative Results• 6 uniformly distributed cameras• 12 degree (RMS) error over 21 joint angles

Pose Estimation Results

Qualitative Results

Pose Estimation Results

Video 1, Camera 1

Pose Estimation Results

Video 1, Camera 2

Pose Estimation Results

Video 2, Camera 1

Pose Estimation Results

Video 2, Camera 2

Future Work

• Use dimensionality reduction to reduce the number of pose parameters.

- results in less number of pose parameters to optimize- would speed up inference

• High resolution reconstruction by a coarse to fine strategy

• Parameter Learning in Object Specific MRF

Thank You

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

+