Inverse Volume Rendering with Material Dictionaries

Inverse Volume Rendering with Material Dictionaries

Ioannis Gkioulekas1 Shuang Zhao2 Kavita Bala2

Todd Zickler1 Anat Levin3

1Harvard 3Weizmann2Cornell

1

Most materials are translucent

2

jewelry

skin

architecture

Photo credit: Bei Xiao, Ted Adelson

food

We know how to render them

3

Monte-Carlo rendering

material parameters

Veach 1997, Dutré et al. 2006

?rendered image

We show how to measure them

4

inverse rendering

material parameters

rendered imagecaptured photograph

Our contributions

5

material1. exact inverse volume rendering

• with arbitrary phase functions!

2. validation with calibration materials known

parameters

3. database of broad range of materialsthin thick

non-dilutable solids

material sample

Why is inverse rendering so hard?

6

radiative transferrandom walk of photons inside

volume • volume light transport has very complex dependence material parameters

thin thick

non-dilutable solids

thin thick

non-dilutable solids

Light transport approximations

7

Previous approach: single-scattering

random walk of photons inside

volume

single-bounce random walk

Narasimhan et al. 2006

Light transport approximations

8

Previous approach: diffusion

Jensen et al. 2001 Papas et al. 2013

…………

isotropic distribution of

photons

parameter ambiguity

≈ ≠material 1

material 2

random walk of photons inside

volume

thin thick

non-dilutable solids

Inverse rendering without approximations

9

random walk of photons inside

volume

exact inversion of random walk

thin thick

non-dilutable solids

Our approach

10

appearance matching

ii. operator-theoretic analysis

i. material representation

iii. stochastic optimization

Background

11

phase function p(θ)scattering coefficient σs

extinction coefficient σt

θ

m = (σt σs p(θ))

random walk of photons inside

medium

Papas et al. 2013

Phase function parameterization

12

g∈ (−1,1 )❑

not general enough

Henyey-Greenstein lobesChen et al. 2006

Donner et al. 2008

Fuchs et al. 2007

Goesele et al. 2004

Gu et al. 2008

Hawkins et al. 2005

Holroyd et al. 2011

McCormick et al. 1981

Pine et al. 1990

Prahl et al. 1993

Wang et al. 2008

Gkioulekas et al. 2013

Narasimhan et al. 2006

Jensen et al. 2001

Previous approach: single-parameter families

m = Σq πq mqp = Σq πq pq

D = {m1, m2, …, mQ}

Dictionary parameterization

13

tent phase functions

D = {p1, p2, …, pQ}

p1p2p3p4p5p6p7p8p9p10p11

dictionary of

• arbitrary

p • similarly for σt and σs

π1

π2

π3π4π5π6π7π8π9π10π11

D

phase functions

phase functions

materials

materials

σt = Σq πq σt,q σs = Σq πq σs,q

Our approach

14

appearance matching

ii. operator-theoretic analysis

i. material representation

iii. stochastic optimization

m = Σq πq mq

Operator-theoretic analysis

15

m = (σt σs p(θ))

τ τ ττ

random walk of photons inside

mediumdiscretized random walk paths• propagation step τ

• total radiance

K(π) = Σq πq Kq

Operator-theoretic analysis

16

m = (σt σs p(θ))

discretized random walk paths• propagation step τ

L(x, θ)

radiance at all medium points and directions

Ln+1(x, θ) = Ln(x, θ)K

• rendering operator R= (I - K)-1 LinputL = Σn Ln

L(x, θ) = R Linput(x, θ)

radiance after n steps

radiance after n+1

steps

R(π)= (I - Σ q πq Kq)-1

dictionary representation:

m = Σq πq mq

Our approach

17

appearance matching

ii. operator-theoretic analysis

i. material representation

iii. stochastic optimization

m = Σq πq mq

R(π)= (I - Σ q πq Kq)-1

Stochastic optimization

18

appearance matching

analytic operator expression for gradient! =

R(π)

render(π)single-stepq ··render(π)R(π)Kq

• gradient descent optimization for inverse rendering

min ǁ photo - render(π) ǁ2

π

Stochastic optimization

19

• exact gradient descent

for k = 1, …, N,

πk = πk - 1 - ak

end

N = a few hundreds

several CPU hours*

=intractable

exact

Stochastic optimization

20

Monte-Carlo rendering to compute

102 samplesnoisy + fast

104 samples 106 samplesaccurate + slow

Stochastic optimization

21

• exact gradient descent

for k = 1, …, N,

πk = πk - 1 - ak

end

N = a few hundreds

several CPU hours*

=intractable

• stochastic gradient descent

for k = 1, …, N,

πk = πk - 1 - ak

end

N = a few hundreds

few CPU seconds*

=solvable

exact

noisy

Theory wrap-up

22

appearance matching

ii. operator-theoretic analysis

i. material representation

iii. stochastic optimization

m = Σq πq mq

R(π)= (I - Σ q πq Kq)-1

𝜕 loss (π )𝜕 π q

noisy

min ǁ photo - render(π) ǁ2

π

Our contributions

23

material1. exact inverse volume rendering

• with arbitrary phase functions!

2. validation with calibration materials known

parameters

3. database of broad range of materialsthin thick

non-dilutable solids

Measurements

24

multiple lighting multiple viewpoints

appearance matching

min ǁ photo - render(π) ǁ2

π

Acquisition setup

25

material sample

frontlighting

backlightingcamera

Acquisition setup

26

bottom rotation stage

top rotation stage

material sample

frontlighting

backlighting

material samplefrontlighting

camera

backlighting

bottom rotation stage

top rotation stage

camera

Validation

27 Frisvad et al. 2007

polystyrenemonodispersions

aluminum oxidepolydispersions

very precise dispersions (NIST Traceable Standards)

calibration materials

known parameters

Mie theory

size

%particle material

medium material

Parameter accuracy

28

polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide

all parameters estimated within 4% error

comparison of ground-truth and measured parameters

ground-truthmeasuredHenyey-Greenstein fit

θ

-π π0

p(θ)

Matching novel measurements

29

captured rendered rendered with HG profilespolystyrene 3

comparison of captured and rendered images

images under unseen geometries predicted within 5% RMS error

ground-truthmeasuredHenyey-Greenstein fit

Our contributions

30

material1. exact inverse volume rendering

• with arbitrary phase functions!

2. validation with calibration materials known

parameters

3. database of broad range of materialsthin thick

non-dilutable solids

thin thick

non-dilutable solids

Measured materials

31

mustard

whole milk

shampoo

hand cream

coffee

wine

robitussin

olive oil curacao

mixed soap

milk soap

liquid clayreduced milk

extinction absorption first momentmaterial R G B R G B R G B

whole milk 100.92 105.345 102.768 0.013 0.013 0.041 0.954 0.963 0.946reduced milk 57.291 62.46 63.757 0.007 0.007 0.024 0.954 0.957 0.942mustard 16.447 18.536 6.457 0.057 0.061 0.451 0.155 0.173 0.351shampoo 8.111 9.919 10.575 0.178 0.328 0.439 0.907 0.882 0.874hand cream 20.82 32.353 41.798 0.011 0.011 0.012 0.188 0.247 0.265liquid clay 37.544 48.25 67.949 0.004 0.004 0.005 0.312 0.442 0.512milk soap 7,625 8.004 8.557 0.003 0.004 0.015 0.164 0.167 0.155mixed soap 3.923 4.018 4.351 0.003 0.005 0.013 0.33 0.322 0.316glycerine soap 0.201 0.202 0.221 0.001 0.001 0.002 0.955 0.949 0.943robitussin 0.009 0.001 0.001 0.012 0.197 0.234 0.906 0.977 0.98coffee 0.054 0.051 0.049 0.275 0.309 0.406 0.911 0.899 0.906olive oil 0.041 0.039 0.012 0.062 0.047 0.353 0.946 0.954 0.975blue curacao 0.01 0.012 0.021 0.083 0.048 0.011 0.955 0.973 0.979red wine 0.015 0.013 0.011 0.122 0.351 0.402 0.947 0.975 0.977

whole milk reduced milk mustard shampoo hand cream

liquid clay milk soap mixed soap glycerine soap robitussin

coffee olive oil curacao wine

Measured phase functions

33

θ

-π π0

p(θ)

measuredHenyey-Greenstein fit

Synthetic images

34

mixed soap

glycerine soap olive oil curacao whole milk

rendered image

Synthetic images

35

chromaticity

Synthetic images

36

mixed soap

glycerine soap olive oil curacao whole milk

rendered image

Effect of phase function

37

mixed soap

measured phase function

Henyey-Greenstein fit

θ-π π0

p(θ)rendered image chromaticity

measuredHenyey-Greenstein fit

Discussion

38

• faster capture and convergence: trade-offs between accuracy, generality, mobility, and usability

• more interesting materials: more general solids, heterogeneous volumes, fluorescing materials

• other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging)

Take-home messages

39

material1. exact inverse volume rendering

• with arbitrary phase functions!

2. validation with calibration materials known

parameters

3. database of broad range of materialsthin thick

non-dilutable solids

Acknowledgements

40

• Henry Sarkas (Nanophase)• Wenzel Jakob (Mitsuba)

Funding:• National Science Foundation • European Research Council• Binational Science Foundation• Feinberg Foundation• Intel• Amazon

http://tinyurl.com/sa2013-inverseDatabase of measured materials:

Error surface

41

appearance matching min ǁ photo - render(π) ǁ2

π

Light generation

42

MEMS light switch

RGB combiner

blue (480 nm) laser

green (535 nm) laser

red (635 nm) laser