Radiometric Compensation through Inverse Light Transport 1|30Wetzstein and Bimber
Radiometric Compensation through Inverse Light Transport
Gordon Wetzstein and Oliver Bimber
Pacific Graphics 2007
contact: [email protected], [email protected]
Radiometric Compensation through Inverse Light Transport 2|30Wetzstein and Bimber
original image observed projection
SmartProjector - no Screens required!
Radiometric Compensation through Inverse Light Transport 3|30Wetzstein and Bimber
live-stage performances
SmartProjector - Applications
museums
cultural heritage sitesarchitectural visualization
outdoor advertisement car interiorair plane cabin
Radiometric Compensation through Inverse Light Transport 4|30Wetzstein and Bimber
SmartProjector - Limitations
no direct mapping: refractions no direct mapping: inter-reflections
Radiometric Compensation through Inverse Light Transport 5|30Wetzstein and Bimber
Related Work
[Yang et al. 2005]
unconventional projections[no screens, HDR, high-speed, super-resolution]
state-of-the-art report: Bimber et al. „The Visual Computing of Projector-Camera Systems“, EG 2007
tiled screen calibration [geometric correction and luminance matching]
book: Majumder and Brown „Practical Multi-Projector Display Design“, AK Peters 2007
image-based relighting, environment matting and dual photography [forward light transport acquisition and relighting]
[Debevec et al. 2000], [Masselus et al. 2003], [Sen et al. 2005], [Zonker et al. 1999]
inverse illumination [indirect light removal for photography and projection]
[Seitz et al. 2005], [Bimber et al. 2006]
focus related projector-camera techniques [image sharpening for defocused projections]
[Bimber and Emmerling 2006], [Zhang and Nayar 2006], [Brown et al. 2006]
Radiometric Compensation through Inverse Light Transport 6|30Wetzstein and Bimber
The 8D Reflectance Field
?
),,,( φϕvufLF =
Radiometric Compensation through Inverse Light Transport 7|30Wetzstein and Bimber
Forward Light Transport
( ) ( ) ( ) ( ) '',',,,,~
wdwxLwwxTwxLwxL ieo ∫Ω+=
( ) ( ) ( ) ( )∑+=j jijiieio wLwxTxLxL ,
0 ( 1)0 0 0 0 0
0 ( 1)( 1) ( 1) ( 1) ( 1) ( 1)
pq
pqmn mn mn pq mn
c T p e
c t t p e
c t t p e
λ λ λ λ
λ λ λ λ λ
λ λ λ λ λ
−
−− − − − −
= +
⎡ ⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥= +⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦⎣ ⎦
LM M O M M M
L
eoiL ,, incoming, outgoing, emissivelight field
~T transport function
T discrete transport function
wx , points in space / discretesamples
', ww directions
cλ camera image eλ environment light in camera space
pλ projected lightλT light transport matrix
λ color channelmn camera resolution pq projector resolution
radiometric compensation?
Radiometric Compensation through Inverse Light Transport 8|30Wetzstein and Bimber
Light Transport Acquisition
m
n
q
p
mn x 1
pq x 1
C
P
pq
mnc = Tp
Radiometric Compensation through Inverse Light Transport 9|30Wetzstein and Bimber
Light Transport Acquisition
m
n
q
p
mn x 1
pq x 1
C
P
pq
mnc = Tp
Radiometric Compensation through Inverse Light Transport 10|30Wetzstein and Bimber
Light Transport Acquisition
m
n
q
p
mn x 1
pq x 1
C
P
pq
mnc = Tp
Radiometric Compensation through Inverse Light Transport 11|30Wetzstein and Bimber
Light Transport Acquisition
m
n
q
p
mn x 1
pq x 1
C
P
pq
mnc = Tp
Radiometric Compensation through Inverse Light Transport 12|30Wetzstein and Bimber
Light Transport Acquisition
m
n
q
p
mn x 1
pq x 1
C
P
pq
mnc = Tp
Radiometric Compensation through Inverse Light Transport 13|30Wetzstein and Bimber
Light Transport Acquisition
projected patterns camera image
video clip
Radiometric Compensation through Inverse Light Transport 14|30Wetzstein and Bimber
Dual Photography
m
n
q
p
mn x 1
pq x 1
P’
C’
pq
mn
c’ = TTp’
c = Tp pq
mn
T
’
’
[Sen et al. 2005]
interchange camera and projector by transposing the light transport matrix
Radiometric Compensation through Inverse Light Transport 15|30Wetzstein and Bimber
dual
light transport matrix T
Dual Photography
composition
illumination pattern
illuminated composition
illuminated dual
Tpc =
'' cTp T=
Radiometric Compensation through Inverse Light Transport 16|30Wetzstein and Bimber
Generalized Radiometric Compensation
R G BR R R R G R B R
R G BG G R G G G B G
R G BB B R B G B B B
c T p T p T p e
c T p T p T p e
c T p T p T p e
= + + +
= + + +
= + + +
R G BR R R R R R
R G BG G G G G G
R G BB B B B B B
c e T T T pc e T T T pc e T T T p
⎡ ⎤−⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥− = ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥−⎣ ⎦ ⎣ ⎦⎣ ⎦
c T p eλ λ λ λ= +
0 0 0 ( 1) 00 0 0 0 0 0
0 0 0 ( 1) 00 0 0 0 0 0
0 0 0 ( 1) 00 0 0 0 0 0
0 0 0 ( 1)( 1) ( 1) ( 1) ( 1) ( 1) ( 1)
R G B k BR R R R R R R
R G B k BG G G G G G G
R G B k BB B B B B B B
R G B k Br B r B r B r B r B r B
c e T T T T pc e T T T T pc e T T T T p
c e T T T T
−
−
−
−− − − − − −
− ⎡ ⎤⎡ ⎤⎢ ⎥⎢ ⎥− ⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥− =⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥−⎣ ⎦ ⎣ ⎦
LLL
M M M M O ML ( 1)k
Bp−
⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
M
single camera, single projector
general setup with r cameras and k projectors
solve with iterative non-negative least squares
Radiometric Compensation through Inverse Light Transport 17|30Wetzstein and Bimber
Diffuse Scattering and Inter-Reflections
Radiometric Compensation through Inverse Light Transport 18|30Wetzstein and Bimber
Diffuse Scattering and Inter-Reflections
[“The
Chu
bbC
hubb
s“, P
ixar
]shadows cannot be compensated with single projector
Radiometric Compensation through Inverse Light Transport 19|30Wetzstein and Bimber
Defocus Compensation
original uncompensated
compensation compensated
Radiometric Compensation through Inverse Light Transport 20|30Wetzstein and Bimber
Multi-projector compensation
left rightleft + right
Radiometric Compensation through Inverse Light Transport 21|30Wetzstein and Bimber
Interactive Compensation on the GPU
reformulate problem for GPU optimized implementation
pre-processing: compute inverse light transporton-line matrix-vector multiplicationSVD:
λλλλ epTc +=
( )
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=−
−−−
+
−−
−
−
+
)1(
0
)1()1(
00
)1()1(
)1(0
0)1(
00
pqmnmnpqmn
pq
mn
p
p
ec
ec
tt
ttpecT
λ
λ
λλ
λλ
λλ
λλ
λλλλ
ΜΜΛ
ΜΟΜΛ
,T TT U V T V U+ += Σ = Σ
Radiometric Compensation through Inverse Light Transport 22|30Wetzstein and Bimber
Sample Light Transport
composition
dual
Radiometric Compensation through Inverse Light Transport 23|30Wetzstein and Bimber
Cluster Decomposition
Radiometric Compensation through Inverse Light Transport 24|30Wetzstein and Bimber
Compensation Results
[“9“,
Focu
s Fe
atur
es a
nd 9
, LLC
]
Radiometric Compensation through Inverse Light Transport 25|30Wetzstein and Bimber
Projecting on Refractive Material
Radiometric Compensation through Inverse Light Transport 26|30Wetzstein and Bimber
Projecting on Refractive Material
light transport matrixpseudo-inverse
Radiometric Compensation through Inverse Light Transport 27|30Wetzstein and Bimber
Projecting on Refractive Material
[“Mike‘s New Car“, Pixar]
Radiometric Compensation through Inverse Light Transport 28|30Wetzstein and Bimber
Interactive Compensation on GPU
[“Mike‘s New Car“, Pixar]30 fps, GeForce 7900 GTX
video clip
Radiometric Compensation through Inverse Light Transport 29|30Wetzstein and Bimber
Summarygeneralized theory of radiometric compensation using inverse light transport
proof-of-concept:– diffuse scattering and inter-reflections– reflecting statuette– refracting glass– defocus compensation– multiple overlapping projector
interactive compensation on the GPU
Radiometric Compensation through Inverse Light Transport 30|30Wetzstein and Bimber
Limitations
projection hardware
physical setup
computational resources
– resolution– black level– brightness | contrast– depth of focus
– environment light– projection surface
– matrix sparsity
Radiometric Compensation through Inverse Light Transport 31|30Wetzstein and Bimber
Outlookview-dependent compensation [Bimber et al. 2005]
incremental inverse light transport acquisition (possibly direct-indirect separation) [Nayar et al. 2006]
novel transport acquisition|storage|processing schemes [Garg et al. 2006]
Radiometric Compensation through Inverse Light Transport 32|30Wetzstein and Bimber
Thank you!Questions?
www.cs.ubc.ca/~wetzste1www.uni-weimar.de/medien/AR
Radiometric Compensation through Inverse Light Transport 33|30Wetzstein and Bimber
Related WorkSeamless Multi-Projections
Inverse Illumination
Forward Light Transport, BRDF Acquisition and Relighting
Focus Related Projector-Camera Techniques
Seitz, S., Matsushita, Y., Kutulakos, K. A Theory of Inverse Light Transport. ICCV, 2005
Ashdown, M., Okabe, T., Sato, I., Sato, Y. Robust Content-Dependent Photometric Projector Compensation, ProCams 2006
Bimber, O., Grundhöfer, A., Zeidler, T., Danch, D., Kapakos, P. Compensating Indirect Scattering for Immersive and Semi-ImmersiveProjection Displays. IEEE VR, 2006
Bimber, O., Emmerling, A. Multi-Focal Projection: A Multi-Projector Technique for Increased Focal Depth. IEEE TVCG, 2006
Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, I., Bolas, M. Synthetic Aperture Confocal Imaging. SIGGRAPH 04
Zhang, L., Nayar, S. Projection Defocus Analysis for Scene Capture and Image Display. SIGGRAPH 06
Masselus, V., Peers, P., Dutré, P, Willems, Y. Relighting with 4D incident Light Fields. ACM TOGS 2003
Goesele, M., Lensch. H., Lang, J., Fuchs, C., Seidel, H. DISCO: Acquisition of Translucent Objects. SIGGRAPH 04
Peers, P., vom Berge, K., Matusik, W., Ramamoorthi, R., Lawrence, J., Rusinkiewicz, S., Dutré, P. A Compact Factored Representation of Heterogeneous Subsurface Scattering. SIGGRAPH 2006Sen, P., Chen, B., Garg, G, Marschner, S, Horowitz, M., Levoy, M., Lensch, H. Dual Photography. SIGGRAPH 05
Debevec, P., Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., Sagar, M. Acquiring the Reflectance Field of a Human Face. SIGGRAPH 00
Yang, R., Majumder, A, Brown, M. Camera Based Calibration Techniques for Seamless Multi-Projector Displays. ACM TOGS. 2005
Nayar, S. Peri, H., Grossberg, M., Belhumeur, P. A Projection System with Radiometric Compensation for Screen Imperfections. ProCams 2003Bimber, O., Emmerling, A., Klemmer, T. Embedded Entertainment with Smart Projectors. IEEE Computer, 2005Bimber, O., Iwai, D., Wetzstein, G., Grundhöfer, A. The Visual Computing of Projector-Camera Systems. EuroGraphics (STAR) 2007Fuji, K., Grossberg, M., Nayar, S. A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments. IEEE CVPR 2005
Grossberg, M., Peri, H., Nayar, S. Making one Object Look Like Another: Controlling Appearance using a Projector-Camera System. IEEE CVPR 2004
Bimber, O., Wetzstein, G., Emmerling, A., Nitschke, C. Enabling View-Dependent Stereoscopic Projection in Real Environments. ISMAR 05
Grundhöfer, A., Bimber, O. Real-Time Adaptive Radiometric Compensation. IEEE Transactions on Visualization and Computer Graphics, to appear
Brown, M., Song, P., Cham, T. Image Pre-Conditioning for Out-of-Focus Projector Blur. CVPR 06
Bimber, O., Iwai, D., Wetzstein, G., Grundhöfer, A., The Visual Computing of Projector-Camera Systems. EuroGraphics state-of-the-art report 2007
Majumder, A, Brown, M. Practical Multi-Projector Display Design. AK Peters 2007
Radiometric Compensation through Inverse Light Transport 34|30Wetzstein and Bimber
Comparison CPU - GPU
pseudo-inverse is less stable than solving explicitly
however, no visible difference
Radiometric Compensation through Inverse Light Transport 35|30Wetzstein and Bimber
Error Analysis