Embed Size (px)
Citation preview
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
An Incremental Weighted Least Squares Approach To Surface Light Fields
Greg Coombe
Anselmo Lastra
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
2
Image-Based Rendering
Generate new views of a scene from existing views
Sample appearance from physical worldLightfields [Levoy96], Lumigraphs [Gortler96]
…
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
3
Surface LightfieldsA surface light field [Wood00] represents the appearance of a model with known geometry and static lighting
( , , , )f u vθ φviewing direction
surface position
(u,v)
(θ,Φ)
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
4
Surface Lightfields
f1(Θ, Φ) f2(Θ, Φ)
f3(Θ, Φ)
f4(Θ, Φ)
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
5
SLF - Batch Process
Surface Lightfield Construction
Renderer
…
[Chen02, Hillesland03]
disk
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
6
SLF - Online Process
IncrementalSurface Lightfield
ConstructionRenderer
1024x768@ 15fps
[Coombe05]
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
7
Scattered Data Approximation
[Coombe05] required resampling to gridSamples are at arbitrary locations in domain due to geometry and camera
?
?
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
8
Scattered Data Approximation
Scattered Data Approximation in lightfields
Unstructured Lightfields [Buehler01]• Tesselation of pure lightfield
Polynomial Texture Maps [Malzbender01] • Fit polynomials to set of images
Radial Basis Functions [Zickler05]• Interpolate sparse reflectance data
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
9
Outline
A 2D scattered data approximation
Fast incremental construction
Realtime evaluation
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
10
Our Approach
Represent the surface lightfield using Weighted Least Squares approximation
Modify WLS for the incremental framework
Adaptive and Hierarchical
CPU/GPU implementationProcess each image in about 1-2 seconds
Real-time rendering
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
11
Bust model,75 training images
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
12
Least Squares Fitting
Find the “best” polynomial approximation to the input samples
“best” means minimizes sum of squared differences
the coefficients are determined by solving a linear system
input samples
reconstruction
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
13
Least Squares
domain
reconstructedfunction
input samples
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
14
Weighted Least Squares
Problem: LS is a global approximation
Solution: Divide domain into multiple LS approximations, and combine to get global approximation
Use a set of low-degree polynomials
Non-linear blending (Partition of Unity)
Good discussion in Scattered Data Approximation, Holgar Wendland
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
15
Weighted Least Squares
centers domains
polynomial approximations
reconstructedfunction
input samples
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
16
Weighted Least Squares
xx
x x
Θ
Φ
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
17
Outline
A 2D scattered data approximation
Fast incremental construction
Realtime evaluation
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
18
Incremental WLS
Feedback is important in SLF construction
As each image is captured, it must be incorporated into the representation
How do we determine the size of each domain?
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
19
Adaptive Construction
x
x
x
x x
x
x
x
Start out with large domainsAdaptive shrink as more points arrive
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
20
Hierarchical Construction
xx
xx
x
xx
x
x
x x
Start out with a single domainSubdivide as more points arrive (quadtree)
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
21
HierarchicalConstruction, First 10 images
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
22
Hierarchical and Adaptive
Hierarchical1x1 Polynomial Texture Mapping
Fast at first, slows down as refines
# of domains is a power of 2
AdaptiveSlow at first, speeds up as domains shrink
Can handle arbitrary # of domains
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
23
Outline
A 2D scattered data approximation
Fast incremental construction
Realtime evaluation
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
24
Implementation
Pose Estimation
Visibility / Reprojection
Incremental Weighted Least Squares
1024x768@ 15fps
GPU Renderer
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
25
Φ
Θ
GPU Implementation
Φ
Θ
Φ
Θ
Φ
Θ
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
26
Results
29K patches
4K patches
side by side, 14K patches
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
27
Pitcher model,65 images
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
28
Performance
~ 0.5 - 2 seconds per image for hierarchical construction
0.5s for 4K bust model
2s for 30K pitcher model
95% is Least Squares FittingAdaptive is 2-3x more expensive
Rendering is 60fps or more
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
29
Conclusion
Represent the surface lightfield using Weighted Least Squares approximation
Modify WLS for the incremental framework
Adaptive and Hierarchical
CPU/GPU implementationProcess each image in about 1-2 seconds
Real-time rendering
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
30
Future Work
Since construction is dominated by LS fitting, implement on GPU
Extend to surface reflectance (4D)Change basis functions
Order-of-magnitude more data
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
31
Thanks
NVIDIA Graduate Student Fellowship
UNC Graphics Group
National Science Foundation
Questions?
