High Resolution Surface Reconstruction from Overlapping
Multiple-Views
Nader Salman Mariette Yvinec Geometrica Team-Project JGA January
29th, 2009 Motivation Urban scenes are hard to model
Complex geometry and topology Severe occlusions We target automatic
surface reconstruction from stereo vision data Related Work
Image-based methods Model-based methods
Reconstructing 3D models of objects and scenes from N-views
Image-based methods Image mosaics [Szeliski et al. 97] Lightfield
[Levoy et al. 96] and Lumigraph [Cohen et al. 96] Model-based
methods Active sensing approach [Turk et al. 94], [Laurendeau et
al. 95], [Levoy et al. 96] Passive sensing approach Silhouette
based [Niem et al. 97] Structure from motion Simple planar scenes
[Zisserman et al. 96, 98] Semi-automatic reconstruction [Debevec et
al. 96] Automatic reconstruction [Boissonnat et al. 88], [Taylor et
al. 03], [Hilton 05], [Pons et al. 07] Image based methods : no
explicit 3D model of the scene, new views are synthesized by
resampling directly the captured images Image mosaics : multiple
overlapping images are matched and integrated into a single
panoramic image Lightfield & Lumigraph : Large number of
calibrated images are captured and resampled to generate new views.
Model based methods : Reconstruction of 3D surface model that can
subsequently be rendered to generate new views Active sensing :
project structured light pattern into the scene to recover dense 3D
measurements of points on the scene surface (Range images : 2D
images + depth per pixel => dense data) Passive sensing :
reconstruct 3D estimates of observed scene features from multiple
camera views (Sparse data) Silhouette based: Use
foreground/background separation to reconstruct object shape from
occluded contours. Structure from motion: Reconstruct 3D estimate
of the scene structure from passive video image sequences. Planar
surface fitting (Simple planar scenes) : planar surfaces can be
fitted to unstructured cloud of3D points. Semi-automatic algorithm
: reconstruct scene model from sparse 3D data # volumetric
primitives placed according to the 3D features of the scene and
constrained optimization reconstructs a best-fit model. Automatic
algorithm : reconstruct automatically scene geometry from sparse 3D
data set. System Overview Image Capture 2D Constrained
Triangulation
3D Triangle Soup Triangle soup Filtering Structure from Motion
Input Preprocessing Reconstruction System Overview Image Capture 2D
Constrained Triangulation
3D Triangle Soup Triangle soup Filtering Structure from Motion
Input Preprocessing Reconstruction 5 Image Capture and 3D recovery
(1)
Video sequences captured images ( images) ~10k points Matchmoving
(camera calibration & point cloud generation)
[MatchMover,Voodoo/VooCAT, or Boujou] Image Capture and 3D recovery
(2)
High resolution localizedstill pictures captured images (10-50
images) Features extraction [ENPC/Certis CSTB] ~300k points Frames
of tracked video [MatchMover, Voodoo/VooCAT, or Boujou] Hardware
[Gyroviz] Image Capture and 3D recovery (2)
High resolution localizedstill pictures captured images (10-50
images) Features extraction [ENPC/Certis CSTB] ~300k points Frames
of tracked video [MatchMover, Voodoo/VooCAT, or Boujou] Hardware
[Gyroviz] 8 Image Capture and 3D recovery (2)
Matchmoving [Voodoo] Output is data dependent - Movement while
shooting video - Lighting conditions Sparse point clouds Outliers
~300k points ~10k points Features extraction [ENPC/Certis CSTB]
Dense point clouds (10-20x more) Requires calibrated images Many
outliers Slow 9 System Overview Image Capture 2D Constrained
Triangulation
3D Triangle Soup Triangle soup Filtering Structure from Motion
Input Preprocessing Reconstruction 10 Input Preprocessing (1)
Outliers removal wrt average K-nearest neighbors squared distance
Detected outliers Detected outliers Input Preprocessing (2)
Outliers removal wrt point error (number of cameras, cameras cone
angle) Detected outliers Detected outliers Input Preprocessing
(3)
3D Points merging Point cloud smoothing (jet-fitting +
reprojection) [Pouget-Cazals07] System Overview Image Capture 2D
Constrained Triangulation
3D Triangle Soup Triangle soup Filtering Structure from Motion
Input Preprocessing Reconstruction 14 Reconstruction
Algorithm
1 Extract 3D triangles that lie on the surface of the scene.
Extract a surface out of these triangles. 2 Triangle soup
extraction (1)
Idea: Extract 3D triangles whose corresponding 2D does not
intersect the image contour N times N images Step 3 2D constrained
Tessellation Backprojected into 3D space Step 4 Step 1 Filter
triangle soup Contour extraction Constraints retrieval Step 2
Triangle soup Triangle soup extraction (2)
Idea: Extract 3D triangles whose corresponding 2D does not
intersect the image contour Step 1: Contour extraction Grayscale
Effective luminance of a pixel Filter Gaussian smoothing mask
Extract contours Frei-chen convolution masks Survey Triangle soup
extraction (3)
Idea: Extract 3D triangles whose corresponding 2D does not
intersect the image contour Step 2: Constraints retrieval Build
complete graph of feature points keep edges that respect the
contour The edge weight depends on number of white pixels Adapt
Needleman-Wunsch [NW1970] Filter collinear and intersecting edges
Give higher priority to short edges Triangle soup extraction
(4)
Idea: Extract 3D triangles whose corresponding 2D does not
intersect the image contour Step 3: 2D constrained tessellation and
backprojection Add constraints in 2D Delaunay tessellation
Backproject into 3D space using corresponding 3D points N times N
images Triangle Soup System Overview Image Capture 2D Constrained
Triangulation
3D Triangle Soup Triangle soup Filtering Structure from Motion
Input Preprocessing Reconstruction 20 Triangle soup filtering
(1)
Problem: Triangle soup contains lots of incorrect triangles Filter
triangles using size heuristic Compute bounding box of triangle
soup Each triangle longest edge is compared to the half diagonal of
the bounding box Filter triangles using visibility constraint
[Taylor03] Empty ray in space between the camera location and 3D
point position. Triangle soup filtering (1)
Before size heuristic After size heuristic 22 Triangle soup
filtering (2)
Problem: Triangle soup contains lots of incorrect triangles Filter
triangles using size heuristic Compute bounding box of triangle
soup Each triangle longest edge is compared to the half diagonal of
the bounding box Filter triangles using visibility constraint
[Taylor03] Empty ray in space between the camera location and 3D
point position. 23 Triangle soup filtering (2)
Before visibility constraint After visibility constraint 24 System
Overview Image Capture 2D Constrained Triangulation
3D Triangle Soup Triangle soup Filtering Structure from Motion
Input Preprocessing Reconstruction 25 Reconstruction (1) Restricted
Delaunay
How do we extract a surface from this polygon soup? Restricted
Delaunay Compute 3D Delaunay triangulation of triangle soup
vertices Compute restricted Delaunay to the union of triangles
Restricted Delaunay reconstruction
Show results Reconstruction (2) OR Restricted Delaunay
Extensions:
How do we extract a surface from this polygon soup? Restricted
Delaunay Triangles respect empty circle condition Non watertight
surface (holes) Extensions: Remesh using restricted Delaunay OR
Scatter a large number of points continuously across the surface of
each triangle Create an implicit surface that approximates the
triangle soup Some results LA_Aerial ~8k points 29 Some results
Lion ~20k points 30 Results Ours [Furokawa et al. 2007] [Strecha et
al. 2006]
[Pons et al. 2009] Ours 31 Future work More data acquisition and
testing for automatic parameter
Improve and complete existing algorithms Triangles soup algorithm
Step 1 : Contour extraction Segmentation techniques [Lvy06]
Histogram for automatic threshold [Asano08] Step 4 : Triangle soup
filtering Some big triangles are correct Use circumscribing circle
to differentiate between detected triangles Project triangles on
respective images Use cross correlation to detect incorrect
triangles Sharp features detection and reconstruction Thank you
Nader Salman
