View
230
Download
2
Tags:
Embed Size (px)
Citation preview
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
Chris BuehlerChris BuehlerMichael Bosse Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS
Steven J. Gortler Steven J. Gortler Harvard University
Michael F. Cohen Michael F. Cohen Microsoft Research
The Image-Based Rendering ProblemThe Image-Based Rendering Problem
• Synthesize novel views from reference Synthesize novel views from reference imagesimages
•Static scenes, fixed lighting
•Flexible geometry and camera configurations
The ULR Algorithm The ULR Algorithm
• Designed to work over a range of Designed to work over a range of image and geometry configurationsimage and geometry configurations
• Designed to satisfy desirable Designed to satisfy desirable propertiesproperties
Geometric Fidelity
# o
f Im
ag
es
VDTMVDTM
LFLFULRULR
Desired Camera
“Light Field Rendering,” SIGGRAPH ‘96
u0
s0
u
s
Desired color interpolated from “nearest cameras”
Desired Camera
“Light Field Rendering,” SIGGRAPH ‘96
u
s
Desired Property #1: Epipole consistency
Desired Camera
“The Scene”
“The Lumigraph,” SIGGRAPH ‘96
u
PotentialArtifact
“The Scene”
“The Lumigraph,” SIGGRAPH ‘96
Desired Property #2: Use of geometric proxy
Desired Camera
“The Lumigraph,” SIGGRAPH ‘96
“The Scene”
Desired Camera
“The Lumigraph,” SIGGRAPH ‘96
“The Scene”
Rebinning
Note: all images are resampled.
Desired Camera
Desired Property #3: Unstructured input images
“The Lumigraph,” SIGGRAPH ‘96
“The Scene”
Desired Property #4: Real-time implementation
Desired Camera
Desired Camera
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
“The Scene”
Occluded
Out of view
Desired Camera
“The Scene”
Desired Property #5: Continuous reconstruction
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
Desired Camera
“The Scene”
θ1
θ2
θ3
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
Desired Camera
“The Scene”
θ1
θ2
θ3
Desired Property #6: Angles measured w.r.t. proxy
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
“The Scene”
Desired Camera
Desired Camera
“The Scene”
Desired Property #7: Resolution sensitivity
Previous WorkPrevious Work
Light fields and LumigraphsLight fields and Lumigraphs•Levoy and Hanrahan, Gortler et al., Isaksen et al.
View-dependent Texture MappingView-dependent Texture Mapping•Debevec et al., Wood et al.
Plenoptic Modeling w/Hand-held Plenoptic Modeling w/Hand-held CamerasCameras•Heigl et al.
Many others…Many others…
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
1.1. Epipole consistencyEpipole consistency
2.2. Use of geometric proxyUse of geometric proxy
3.3. Unstructured inputUnstructured input
4.4. Real-time implementationReal-time implementation
5.5. Continuous reconstructionContinuous reconstruction
6.6. Angles measured w.r.t. proxyAngles measured w.r.t. proxy
7.7. Resolution sensitivityResolution sensitivity
Desired Camera
Blending FieldsBlending Fields
colordesired = Σ wi colorii
Desired Camera
Blending FieldsBlending Fields
colordesired = Σ w(ci) colorii
• Explicitly construct blending fieldExplicitly construct blending field•Computed using penalties
•Sample and interpolate over desired image
• Render with hardwareRender with hardware•Projective texture mapping and alpha
blending
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
Angle PenaltyAngle Penalty
Geometric Proxy
θ1
θ2 θ3
θ4
θ5
θ6
penaltyang(Ci) = θi
C1
C2C3
C4
C5
C6
Cdesired
Resolution PenaltyResolution Penalty
Cdesired
Geometric Proxy
penaltyres(Ci) = max(0,dist(Ci) – dist(Cdesired ))
distdesired
pena
lty
res
Ci
Field-Of-View PenaltyField-Of-View Penalty
angle
pena
lty
FO
V
Total PenaltyTotal Penalty
penalty(Ci) =α penaltyang(i) + β penaltyres(i) + γ penaltyfov(i)
K-Nearest Continuous BlendingK-Nearest Continuous Blending
• Only use cameras with K smallest Only use cameras with K smallest penaltiespenalties
• CC0 0 Continuity: contribution drops to Continuity: contribution drops to zero as camera leaves zero as camera leaves K-nearest K-nearest setset
• w(Ci) = 1- penalty(Ci)/penalty(Ck+1st closest )
• Partition of Unity: normalize Partition of Unity: normalize
• w(Ci) = w(Ci)/Σw(Cj)
~
jj
Blending Field VisualizationBlending Field Visualization
Sampling Blending FieldsSampling Blending Fields
Just epipole samplingEpipole and grid sampling
Select blending field sample locationsfor each sample location j do
for each camera i doCompute penalty(i) for sample location j
end forFind K smallest penaltiesCompute blending weights for sample location j
end forTriangulate sample locations
Hardware Assisted AlgorithmHardware Assisted Algorithm
Clear frame bufferfor each camera i do
Set current texture and projection matrixCopy blending weights to vertices’ alpha channelDraw triangles with non-zero alphas
end for
Sample Blending Field
Render with Graphics Hardware
Blending over one triangleBlending over one triangle
Just epipole samplingEpipole and grid sampling
Select blending field sample locationsfor each sample location j do
for each camera i doCompute penalty(i) for sample location j
end forFind K smallest penaltiesCompute blending weights for sample location j
end forTriangulate sample locations
Hardware Assisted AlgorithmHardware Assisted Algorithm
Clear frame bufferfor each camera i do
Set current texture and projection matrixCopy blending weights to vertices’ alpha channelDraw triangles with non-zero alphas
end for
Sample Blending Field
Render with Graphics Hardware
DemoDemo
Future WorkFuture Work
• Optimal sampling of the camera Optimal sampling of the camera blending fieldblending field
• More complete treatment of More complete treatment of resolution effects in IBRresolution effects in IBR
• View-dependent geometry proxiesView-dependent geometry proxies• Investigation of geometry vs. Investigation of geometry vs.
images tradeoffimages tradeoff
ConclusionsConclusions
Unstructured Lumigraph Unstructured Lumigraph RenderingRendering •unifies view-dependent texture mapping and
lumigraph rendering methods
•allows rendering from unorganized images
• sampled camera blending field
AcknowledgementsAcknowledgementsThanks to the members of theThanks to the members of the
MIT Computer Graphics GroupMIT Computer Graphics Groupandand
Microsoft ResearchMicrosoft ResearchGraphics and Computer Vision GroupsGraphics and Computer Vision Groups
• DARPA ITO Grant F30602-971-0283
• NSF CAREER Awards 9875859 & 9703399
• Microsoft Research Graduate Fellowship Program
• Donations from Intel Corporation, Nvidia, andMicrosoft Corporation