View
32
Download
0
Category
Tags:
Preview:
DESCRIPTION
Appearance modeling: textures and IBR Class 17. 3D photography course schedule. Papers. http://www.unc.edu/courses/2004fall/comp/290b/089/papers/. Projects. Volumetric 3D integration. Multiple depth images. Volumetric integration. Appearance Modeling. Texturing Single image - PowerPoint PPT Presentation
Citation preview
Appearance modeling: textures and IBR
Class 17
3D photography course schedule
Introduction
Aug 24, 26 (no course) (no course)
Aug.31,Sep.2
(no course) (no course)
Sep. 7, 9 (no course) (no course)
Sep. 14, 16 Projective Geometry Camera Model and Calibration
(assignment 1)
Feb. 21, 23 Camera Calib. and SVM Feature matching(assignment 2)
Feb. 28, 30 Feature tracking Epipolar geometry(assignment 3)
Oct. 5, 7 Computing F Triangulation and MVG
Oct. 12, 14 (university day) (fall break)
Oct. 19, 21 Stereo Active ranging
Oct. 26, 28 Structure from motion SfM and Self-calibration
Nov. 2, 4 Shape-from-silhouettes Space carving
Nov. 9, 11 3D modeling Appearance Modeling Nov.12 papers(2-3pm SN115)
Nov. 16, 18 (VMV’04) (VMV’04)
Nov. 23, 25 papers & discussion (Thanksgiving)
Nov.30,Dec.2
papers & discussion papers and discussion Dec.3 papers(2-3pm SN115)
Dec. 7? Project presentations
PapersLi Exact Voxel Occupancy with Graph Cuts
Sudipta Stereo without epipolar lines
ChrisA graph cut based adaptive structured light approach for real-time range acquisition
Nathan Space-time faces
Brian Depth-from-focus …
ChadInteractive Modeling from Dense Color and Sparse Depth
Seon Joo Outdoor calibration of active cameras
Jason spectral partitioning
Sriram Linear multi-view reconstruction
Christine 3D photography using dual …
http://www.unc.edu/courses/2004fall/comp/290b/089/papers/
Projects
Chris Wide-area display reconstruction
Nathan Structured light
Brian Depth-from-focus/defocus
Li Visual-hulls with occlusions
Chad Laser scanner for 3D environments
Seon Joo Collaborative 3D tracking
Jason SfM for long sequences
SudiptaCombining exact silhouettes and photoconsistency
Sriram Panoramic cameras self-calibration
Christine desktop lamp scanner
Multiple depth images Volumetric integration
Volumetric 3D integration
Appearance Modeling
• Texturing• Single image• Multiple image
• Image-based rendering• (Unstructured) lightfield rendering• Surface lightfields
Texture mapping 3D model
Need to estimate relative pose between camera and 3D model
Texture Mapping
• Conventional texture-mapping with texture coordinates
• Projective texture-mapping
Texture Map Synthesis I• Conventional Texture-
Mapping with Texture Coordinates• Create a triangular
texture patch for each triangle
• The texture patch is a weighted average of the image patches from multiple photographs
• Pixels that are close to image boundaries or viewed from a grazing angle obtain smaller weights
Photograph
Texture Map
3D Triangle
Texture Map Synthesis II• Allocate space for texture patches
from texture maps• Generalization of memory allocation
to 2D• Quantize edge length to a power of 2• Sort texture patches into decreasing
order and use First-Fit strategy to allocate space
First-Fit
A Texture Map Packed with Triangular Texture Patches
Appearance Modeling
texture atlas
Dealing with auto-exposure
Photometric alignment of textures (or HDR textures)
(Kim and Pollefeys, CVPR’04)
Image as texture
Depth image Triangle mesh Texture image
Textured 3DWireframe model
Affine vs. projective texture mapping (see later)
Lightfield literature
• Plenoptic function
• Lightfield (plane) and Lumigraph (some
geometry)
• Unstructered lightfield (some (view-dependent)
geometry)
• Surface lightfields (full geometry)
• Plenoptic sampling (trade-off geometry vs. images)
(Levoy&Hanrahan,Siggraph´96 Gortler et al.,Siggraph´96)
(Koch et al. ICCV´99; Heigl et al. DAGM´99; Buehler et al. Siggraph‘01)
(Chai et al.,Siggraph´00)
(Wood et al.,Siggraph´00, Chen et al., Siggraph‘02)
(Adelson&Bergen´91; McMillan&Bishop,Siggraph´95)
Lightfield rendering
focal surface
Approximate light rays by interpolating from closest light rays in lightfield
viewpoint surface
• Projection of viewpoint surface in virtual camera determines which views to get lightrays from
• Transfer from images to virtual views over focal surface determines which pixels to use
Unstructured lightfield rendering
original viewpoints
Novel view
For every pixel, combine For every pixel, combine best best rays from rays from closestclosest views views
(Koch et al.,ICCV´99; Heigl et al.,DAGM´99)
Focal surfaceFocal surface
demo
Example: desk sequence
186 images recorded with hand-held camera
Example: desk sequence
structure and motion
depth images
190 images
7000
poin
ts
Example: Desk Lightfield
Planar focal surface
(shadow artefacts)
View-dependent geometry approximation
original viewpoints
object surface
View-dependent surface approximation
Novel view
depth mapsdepth maps
Adaptation of geometry with the rendering viewpoint
View-dependent geometry approximation
Geometry subdivision
original viewpoints
object surface
View-dependent surface approximation
Novel view
depth mapsdepth maps
Note: Only necessary when depth value significantly Note: Only necessary when depth value significantly deviates from previous approximationdeviates from previous approximation
Viewpoint-geometry without subdivision
4 subdivisions
2 subdivisions
1 subdivision of viewpoint surface
Scalable geometric approximation
Example: Desk lightfield
Planar focal surface View-dependent geometry approximation(2 subdivisions)
Hardware accelerated rendering
Use blending operation similar to Gouraud shading
Use projective textures!
Demo
demo
Extrapolation(Buehler et al., Siggraph´01)
Add mesh to cover whole image (compute non-binary blending weights)
Rendered image
Blending field(courtesy Leonard McMillan)
,,,,, srfBGRI
Surface Lightfields
Surface locationSurface location Viewing directionViewing direction
Surface light field (SLF) function
Chen et al., Siggraph 2002, "Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields" R. Grzeszczuk, Presentation on Light Field Mapping, SIGGRAPH 2002 Course Notes for Course “Image-based Modeling.” http://www.intel.com/research/mrl/research/lfm/
Surface Lightfields
• Partition SLF across surface primitives Pi
• Approximate SLF for each Pi individually as
• Surface light field (SLF) function
,*,,,,1
iii Pk
K
k
Pk
P hsrgsrf
Light field maps:Light field maps:stored as 2D texture mapsstored as 2D texture maps
Surface mapsSurface mapsView mapsView maps
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Data Acquisition
• 200-400 images captured by hand-held camera
• Geometry scanned with structured lighting
• Images registered to geometry
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Partitioning• Partitioning the light field data across
small surface primitives• Individual parts add up to original SLF• Ensure continuous approximations
across neighbouring surface elements
Triangle-centered:Triangle-centered:split the light field between split the light field between individual trianglesindividual triangles
Partitioning
Triangle-centered: Triangle-centered: split the light field split the light field between individual between individual trianglestriangles ->discontinuity ->discontinuity
• Partitioning the light field data across small surface primitives• Individual parts add up to original SLF• Ensure continuous approximations
across neighbouring surface elements
Vertex-centered Partitioning
• Partition surface light field data around every vertex
Hat functionHat function
ppj sr
v,
0
a
outside
inside
ring
ring 10 a==
Vertex-centered Partitioning
Vertex-centered Partitioning
• Define local reference frame of the vertex
• Reparameterize each vertex light field to its local coordinate system
jjjj vvvv srfsrf ,,,,,,
Vertex light fieldVertex light field
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Resampling• Goal: Generate vertex light field
function• Visibility computation determines
unoccluded views for each triangle ring• 2 steps:
• Normalization of texture size• Resampling of viewing directions
Resampling
Each column represents a different view
j
i
jjjj vC
vvvv ffffF ...,, 32111stst view view 22ndnd view view
CCii-th view-th view
Resampling
• 1. Normalization of texture size• Each texture patch has the same
shape and size• Bilinear interpolation
• 2. Resampling of viewing directions
Resampling
1. Normalization of texture size2. Resampling of viewing directions
Projection of original viewsProjection of original views
1 2 3 4 ……. c ….. 1 2 3 4 ……. c ….. CCii11
22
……
……
..
mm
MM
Resampling
1. Normalization of texture size2. Resampling of viewing directions
Delaunay triangulationDelaunay triangulationUniform grid of viewsUniform grid of views
MM
1 2 3 4 ……. c ….. 1 2 3 4 ……. c ….. CCii11
22
……
……
..
mm
Resampling
1. Normalization of texture size2. Resampling of viewing directions
1 2 3 4 ……. n ….. 1 2 3 4 ……. n ….. NN11
22
……
……
..
mm
MM
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Decomposition & Approximation
• Rearrange 4-dimensional F into M*N matrix
• Decompose F using matrix factorization• Truncate the sum after K terms
iiij PN
PPv fffF ..., 21
1k
Tkk
v vuF j
N qqvk
K
kpp
vk
jj hsrg ,*,1
1
~
k
Tkk
v
vuFj
K<<NK<<N
K
Surface mapsSurface maps View mapsView maps
Decomposition & Approximation
• Split surface maps for triangle ring into surface maps for individual triangles
• 3 surface maps for each approximation term of each triangle
Decomposition & Approximation
11stst approximation approximation
22ndnd approximation approximation
KKthth approximation approximation
……..
Each approximationEach approximation=3 surface maps + 3 view maps=3 surface maps + 3 view maps
ff
Approximation methods
• PCA (principal component analysis)• Progressive• Arbitrary sign factors
• NMF (non-negative matrix factorization)• Parts-based representation• Non-negative factors• Easier and faster rendering
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Light Field Mapping
Tiled surface mapsTiled surface maps Tiled view mapsTiled view maps
• Light field maps are redundant• Very high compression ratio (10000:1)
Light Field Mapping
Data AcquisitionData Acquisition
ResamplingResampling
PartitioningPartitioning
RenderingRendering
ApproximationApproximation
CompressionCompression
Rendering
11stst approximation approximation
22ndnd approximation approximation
KKthth approximation approximation
……..
Each approximationEach approximation=3 surface maps + 3 view maps=3 surface maps + 3 view maps
ff
Rendering
• Surface map: view-independent
• View map: • Establish vertex
coordinate system
• Project viewing vector onto view map hemisphere
Results
• Bust
• Star
• Turtle• Buddha• Horse
Recommended