View
219
Download
3
Tags:
Embed Size (px)
Citation preview
A Signal-Processing Framework A Signal-Processing Framework for Forward and Inverse Renderingfor Forward and Inverse Rendering
COMS 6998-3, Lecture 8
Illumination IllusionIllumination Illusion
People perceive materials more easily under natural illumination than simplified illumination.
Images courtesy Ron Dror and Ted Adelson
Illumination IllusionIllumination Illusion
People perceive materials more easily under natural illumination than simplified illumination.
Images courtesy Ron Dror and Ted Adelson
Material RecognitionMaterial Recognition
Photographs of 4 spheres in 3 different
lighting conditions
courtesy Dror and Adelson
Estimating BRDF and LightingEstimating BRDF and Lighting
Photographs
Geometric model
InverseRenderingAlgorithm
Lighting
BRDF
Estimating BRDF and LightingEstimating BRDF and Lighting
Photographs
Geometric model
ForwardRenderingAlgorithm
Rendering
Lighting
BRDF
Estimating BRDF and LightingEstimating BRDF and Lighting
Photographs
Geometric model
ForwardRenderingAlgorithm
Rendering
Novel lighting
BRDF
Inverse Problems: Difficulties Inverse Problems: Difficulties
Angular width of Light Source
Su
rfac
e ro
ugh
nes
sIll-posed
(ambiguous)
MotivationMotivation
Understand nature of reflection and illumination
Applications in computer graphics• Real-time forward rendering• Inverse rendering
Contributions of ThesisContributions of Thesis
1. Formalize reflection as convolution
2. Signal-processing framework
3. Practical forward and inverse algorithms
OutlineOutline
• Motivation
• Reflection as Convolution• Preliminaries, assumptions• Reflection equation, Fourier analysis (2D)• Spherical Harmonic Analysis (3D)
• Signal-Processing Framework
• Applications
• Summary and Implications
AssumptionsAssumptions• Known geometry
Laser range scanner
Structured light
AssumptionsAssumptions
• Known geometry
• Convex curved surfaces: no shadows, interreflection
Complex geometry: use surface normal
AssumptionsAssumptions
• Known geometry
• Convex curved surfaces: no shadows, interreflection
• Distant illumination
Photograph of mirror sphere Illumination: Grace Cathedral courtesy Paul Debevec
AssumptionsAssumptions
• Known geometry
• Convex curved surfaces: no shadows, interreflection
• Distant illumination
• Homogeneous isotropic materials
Isotropic Anisotropic
AssumptionsAssumptions
• Known geometry
• Convex curved surfaces: no shadows, interreflection
• Distant illumination
• Homogeneous isotropic materials
Later, practical algorithms: relax some assumptions
ReflectionReflection
2
2
id( )iL Lighting
L
i
( )oB Reflected Light Field
( , )i o BRDF
B
o
Reflection as Convolution (2D)Reflection as Convolution (2D)
2
2
id( )iL Lighting
( )oB Reflected Light Field
( , )i o BRDF
Li o B
i
B
L L
o
Reflection as Convolution (2D)Reflection as Convolution (2D)
2
2
id( )iL Lighting
( )oB Reflected Light Field
( , )i o BRDF
( )iL ( , )i o id2
2
( , )oB
Li o B
L
i
Bo
Reflection as Convolution (2D)Reflection as Convolution (2D)
( )iL ( , )i o id2
2
( , )oB
Li o B
L
i
Bo
ConvolutionConvolution
x
Signal f(x) Filter g(x)
Output h(u)
u
ConvolutionConvolution
x
Signal f(x) Filter g(x)
Output h(u)
u
1 1( ) ( ) ( )h u g x u f x dx
u1
ConvolutionConvolution
x
Signal f(x) Filter g(x)
Output h(u)
u
u2
2 2( ) ( ) ( )h u g x u f x dx
ConvolutionConvolution
x
Signal f(x) Filter g(x)
Output h(u)
u
u3
3 3( ) ( ) ( )h u g x u f x dx
ConvolutionConvolution
x
Signal f(x) Filter g(x)
Output h(u)
u
( ) ( ) ( )h u g x u f x dx
h f g g f Fourier analysis
h f g
Reflection as Convolution (2D)Reflection as Convolution (2D)
B L
Frequency: product
Spatial: integral
( )iL ( , )i o id2
2
( , )oB
, ,2l p l l pB L Fourier analysis
R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208
L
i
Bo
Li o B
Related WorkRelated Work
• Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99
• Reflection as frequency-space operator: D’Zmura 91
• Lambertian reflection is convolution: Basri Jacobs 01
Our Contributions• Explicitly derive frequency-space convolution formula• Formal quantitative analysis in general 3D case
Spherical HarmonicsSpherical Harmonics
-1-2 0 1 2
0
1
2
.
.
.
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
l
m
1
Spherical Harmonic AnalysisSpherical Harmonic Analysis
, ,2l p l l pB L
2D:
22
0 0 ,( , , , ) ( [ , ]) ( , , , )o o i i i i o o i iB L R d d
3D:
( )iL ( , )i o id2
2
( , )oB
, ,lm pq l lm lq pqB L
OutlineOutline
• Motivation
• Reflection as Convolution
• Signal-Processing Framework• Insights, examples• Well-posedness of inverse problems
• Applications
• Summary and Implications
Insights: Signal ProcessingInsights: Signal Processing
Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
Insights: Signal ProcessingInsights: Signal Processing
Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
Filter is Delta function : Output = Signal
Mirror BRDF : Image = Lighting [Miller and Hoffman 84]
Image courtesy Paul Debevec
Insights: Signal ProcessingInsights: Signal Processing
Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
Signal is Delta function : Output = Filter
Point Light Source : Images = BRDF [Marschner et al. 00]
Am
plit
ude
Frequency
Roughness
Phong, Microfacet ModelsPhong, Microfacet Models
Mirror
Illumination estimationill-posed for rough surfaces
Analytic formulae in R. Ramamoorthi and P. Hanrahan“A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128
LambertianLambertian
Incident radiance (mirror sphere)
Irradiance (Lambertian)
N
21
2
2
(1)!2
(2)(1)2!
l
lll
lleven
ll
l
2 / 3
/ 4
0
l0 1 2
R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp 2448-2459
R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp 383-390
Inverse LightingInverse Lighting
Well-posed unless denominator vanishes• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors low pass filters: Inverse lighting ill-
posed
,
,
1 lm pqlm
l lq pq
BL
, ,lm pq l lm lq pq
B L
B L
Given: B,ρ find L
Inverse BRDFInverse BRDF
Well-posed unless Llm vanishes• Lighting should have sharp features (point sources, edges)• BRDF estimation ill-conditioned for soft lighting
,,
1 lm pqlq pq
l lm
B
L
Directional
Source
Area source
Same BRDF
Given: B,L find ρ
Factoring the Light FieldFactoring the Light Field
Light Field can be factored• Up to global scale factor• Assumes reciprocity of BRDF• Can be ill-conditioned• Analytic formula derived
Given: B find L and ρ
B L
4D 2D 3D
More knowns (4D)than unknowns (2D/3D)
OutlineOutline
• Motivation
• Reflection as Convolution
• Signal-Processing Framework
• Applications• Forward rendering (convolution)• Inverse rendering (deconvolution)
• Summary and Implications
Computing IrradianceComputing Irradiance• Classically, hemispherical integral for each pixel
• Lambertian surface is like low pass filter
• Frequency-space analysis
IncidentRadiance
Irradiance
9 Parameter Approximation9 Parameter Approximation
Exact imageOrder 01 term
(constant)
RMS error = 25 %
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
l
m
1
9 Parameter Approximation9 Parameter Approximation
Exact imageOrder 14 terms(linear)
RMS Error = 8%
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
l
m
1
9 Parameter Approximation9 Parameter Approximation
Exact imageOrder 29 terms
(quadratic)
RMS Error = 1%
For any illumination, average error < 2% [Basri Jacobs 01]
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
l
m
1
ComparisonComparison
Incident illumination
300x300
Irradiance mapTexture: 256x256
HemisphericalIntegration 2Hrs
Irradiance mapTexture: 256x256
Spherical HarmonicCoefficients 1sec
Time 300 300 256 256 Time 9 256 256
Inverse RenderingInverse Rendering
Miller and Hoffman 84
Marschner and Greenberg 97
Sato et al. 97
Dana et al. 99
Debevec et al. 00
Marschner et al. 00
Sato et al. 99
Lighting
BRDF
Known Unknown
Unknown
Known
Textures are a third axis
ContributionsContributions
• Complex illumination
• Factorization of BRDF, lighting (find both)
• New representations and algorithms
• Formal study of inverse problems (well-posed?)
ComplicationsComplications
• Incomplete sparse data (few photographs)
• Concavities: Self Shadowing
• Spatially varying BRDFs
ComplicationsComplications
Challenge: Incomplete sparse data (few photographs) Difficult to compute frequency spectra
Solution:• Use parametric BRDF model• Dual angular and frequency space representation
Algorithm ValidationAlgorithm ValidationPhotograph
Kd 0.91
Ks 0.09
μ 1.85
0.13
“True” values
Algorithm ValidationAlgorithm Validation
Renderings
Unknown lighting
Photograph
Known lighting
Kd 0.91 0.89 0.87
Ks 0.09 0.11 0.13
μ 1.85 1.78 1.48
0.13 0.12 0.14
“True” values
Image RMS error 5%
Inverse BRDF: SpheresInverse BRDF: Spheres
Bronze Delrin Paint Rough Steel
Photographs
Renderings(Recovered
BRDF)
ComplicationsComplications
Challenge: Complex geometry with concavities Self shadowing
Solution:• Use associativity of convolution• Blur lighting, treat specular BRDF term as mirror• Single ray for shadowing, easy in ray tracer
Complex GeometryComplex Geometry
3 photographs of a sculpture• Complex unknown illumination• Geometry known• Estimate microfacet BRDF and distant lighting
ComparisonComparison
Photograph Rendering
New View, LightingNew View, Lighting
Photograph Rendering
ComplicationsComplications
Challenge: Spatially varying BRDFs
Solution:• Use textures to modulate BRDF parameters
Textured ObjectsTextured Objects
Photograph Rendering
SummarySummary
• Reflection as convolution
• Frequency-space analysis gives many insights
• Practical forward and inverse algorithms
• Signal-Processing: A useful paradigm for forward and inverse rendering in graphics and vision