Image-space Control Variatesfor Rendering

Fabrice Rousselle1 Wojciech Jarosz2 JanNovák1

1DisneyResearch2DartmouthCollege

SPATIO-TEMPORAL COHERENCE

2

OriginalscenebyWig42,downloadedfromblendswap.com

64spp

SPATIO-TEMPORAL COHERENCE

3

OriginalscenebyWig42,downloadedfromblendswap.com

96kspp

SPATIO-TEMPORAL COHERENCE

4

OriginalscenebyWig42,downloadedfromblendswap.com

96kspp

SPATIO-TEMPORAL COHERENCE

5

OriginalscenebyWig42,downloadedfromblendswap.com

96kspp

SPATIO-TEMPORAL COHERENCE

6

OriginalscenebyWig42,downloadedfromblendswap.com

96kspp

SPATIO-TEMPORAL COHERENCE

7

OriginalscenebyWig42,downloadedfromblendswap.com

96kspp

• Irradiance caching[Ward et al. 1988, …]

• Photon mapping[Jensen 1995, …]

• Image-space denoising[Rushmeier and Ward 1994, …]

• Many more…

PREVIOUS WORKS

8

PREVIOUS WORKS

9

Consistent scene editing [CSE], Günther and Grosch, EGSR 2015

Re-render

1024 spp

Previous Difference

64 spp

PREVIOUS WORKS

10

Consistent scene editing [CSE], Günther and Grosch, EGSR 2015

Re-render

1024 spp

Previous

Gradient-domain path tracing [GDPT], Kettunen et al., SIGGRAPH 2015

16 spp

Noisy

16 spp

Difference (DY)

16 spp

Difference (DX) Poisson solver

Difference

64 spp

OUR WORK

11

Scene editing

1024 spp

Previous

Gradient-domain rendering

16 spp

Noisy

• Unified framework• Control variates• Provably optimal

variance

INTEGRATION WITH CONTROL VARIATES

12

! 𝑓 𝑥 𝑑𝑥&

INTEGRATION WITH CONTROL VARIATES

13

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

Controlvariate withknownintegral

INTEGRATION WITH CONTROL VARIATES

14

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

INTEGRATION WITH CONTROL VARIATES

15

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

INTEGRATION WITH CONTROL VARIATES

16

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

INTEGRATION WITH CONTROL VARIATES

17

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

SCENE EDITING

18

Previous

1024 spp

Re-render

64 spp

SCENE EDITING

19

Re-render

64 spp

Previous

1024 spp

𝐺! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

20

Re-render

Previous

64 spp

SCENE EDITING

1024 spp

𝐺

𝑓

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

SCENE EDITING

21

Previous

Re-render

64 spp

64 spp

1024 spp

𝐺

𝑔

𝑓

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

SCENE EDITING

22

Previous

64 spp

64 spp

1024 spp

𝐺

𝑔

𝑓

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

Re-render same random seed

SCENE EDITING

23

Previous

Difference

64 spp

1024 spp

64 spp

𝑓

𝑔

𝐺

Re-render same random seed

SCENE EDITING

24

Previous

Re-render twicesame random seed

Reuse

Re-render

Difference

64 spp

1024 spp

64 spp

𝑓

𝐺

Reuse

Re-render

OPTIMAL COMBINATION OF ESTIMATORS

25

Re-render

×

Weight

×

Weight

Weight

Weight

WeightWeight Reuse

OPTIMAL COMBINATION OF ESTIMATORS

26

Re-render

×

×

Weight

Weight

Weight

Weight Reuse

OPTIMAL COMBINATION OF ESTIMATORS

27

Weight

Weight

=

=

Optimal

OPTIMAL COMBINATION OF ESTIMATORS

28

Weight

Weight

?×

×

Re-render

Reuse

Optimal

Limitations- Assumes hi-quality

previous image- User-defined threshold- Selection discards data

PREVIOUS WORK: SELECTION HEURISTIC [GG15]

29

Re-render

Reuse

Use if < τ

Use otherwise

Difference

OPTIMAL COMBINATION OF ESTIMATORS

30

Re-render

ReuseIndependent

Re-render

ReuseVariance

Variance

-1

-1

=

Weight

=

Weight

OPTIMAL COMBINATION OF ESTIMATORS

31

Re-render

ReuseIndependent

Variance

Variance

-1

=

Weight

-1

=

Weight

-1

Covariance

CovarianceVariance

Variance

Correlated

2x2 matrixper pixel

Weight

Weight

COMPARISON TO PREVIOUS WORK [GG15] – relative MSE

32

Re-renderPrevious

1024 spp – HQ 64 spp – 0.363

COMPARISON TO PREVIOUS WORK [GG15] – relative MSE

33

Re-renderPrevious

64 spp – 0.363

[GG15] Ours

0.230 0.0841024 spp – HQ

COMPARISON TO PREVIOUS WORK [GG15] – relative MSE

34

Re-renderPrevious

64 spp – 0.363

[GG15] Ours

0.230 0.084

0.384 0.15364 spp – LQ 64 spp – 0.363

1024 spp – HQ

GRADIENT-DOMAIN PATH TRACING

35

16 spp

Throughput

16 spp

Horizontal gradient

16 spp

Vertical gradient

PREVIOUS WORK: POISSON RECONSTRUCTION

36

Poisson solver (L2)

16 spp

Throughput

16 spp 16 spp

Horizontal gradient Vertical gradient

PREVIOUS WORK: POISSON RECONSTRUCTION

37

Poisson solver (L1)

16 spp

Throughput

16 spp 16 spp

Horizontal gradient Vertical gradient

EXTENSION TO MULTIPLE ESTIMATORS: GDR

Hori. gradient

Vert. gradient

DX

DY

TH0 + DX

TH0 - DX

TH0 + DY

TH0 - DY

Throughput

TH0

Left neighbor

Right neighbor

Bottom neighbor

Top neighbor

TH0

None

Control Variate

Hori. gradient

Vert. gradient

EXTENSION TO MULTIPLE ESTIMATORS: GDR

Hori. gradient

Vert. gradient

DX

DY

TH0

TH0 + DX

TH0 - DX

TH0 + DY

TH0 - DY

Hori. gradient

DX

Vert. gradient

DY

Throughput

TH0

Throughput

TH1

AVG . . .

TH1

TH1 + DX

TH1 - DX

TH1 + DY

TH1 - DY

TH50

Throughput [CVPT-uni]

39

EXTENSION TO MULTIPLE ESTIMATORS: GDR

Hori. gradient

Vert. gradient

DX

DY

TH0

TH0 + DX

TH0 - DX

TH0 + DY

TH0 - DY

Hori. gradient

DX

Vert. gradient

DY

Throughput

TH0

Throughput

TH1

. . .

TH1

TH1 + DX

TH1 - DX

TH1 + DY

TH1 - DY

TH50

Throughput [CVPT-opt]

40

TH50

Throughput [CVPT-uni]

OPT

RELATION TO POISSON SOLVER

41

TH50 uniform weights

Iterated reconstruction (Ours)L2Poisson Solver [GDPT]

≈

COMPARISON TO PREVIOUS WORK [GDPT] – relative MSE

42

ReferenceInput

0.0282

OursCVPT-uniGDPT-L2

0.0034 0.0034

OursCVPT-wgtGDPT-L1

0.0032 0.0018

Sponza scene – 16 spp

COMPARISON TO PREVIOUS WORK [GDPT] – relative MSE

43

Bookshelf scene – standard path tracing 0.0590

COMPARISON TO PREVIOUS WORK [GDPT] – relative MSE

44

Bookshelf scene – L1 Poisson reconstruction 0.0094

COMPARISON TO PREVIOUS WORK [GDPT] – relative MSE

45

Bookshelf scene – Ours with optimized weights 0.0044

MORE IN THE PAPER: UNBIASED RECONSTRUCTIONS

46

• Integration with control variates…• simple yet powerful tool• relatively unexplored in rendering• should be used with optimal weighting scheme

• Future work• animations, stereo rendering, light fields• better sampling of the difference buffer

CONCLUSION

47

48

RELATION TO POISSON SOLVER

49

TH50 optimized weights

Iterated reconstruction (Ours)L1Poisson Solver [GDPT]

INTEGRATION WITH CONTROL VARIATES

50

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

INTEGRATION WITH CONTROL VARIATES

51

! 𝑓 𝑥 − 𝑔 𝑥 𝑑𝑥 + 𝐺&

COMPARISON TO PREVIOUS WORK [GDPT] – relative MSE

52

ReferenceInput

0.206

OursCVPT-uniGDPT-L2

0.0365 0.0362

OursCVPT-wgtGDPT-L1

0.0314 0.0129

Bookshelf scene – 256 spp

COMPARISON TO PREVIOUS WORK [GDPT] – relative MSE

53

ReferenceInput

0.0616

OursCVPT-uniGDPT-L2

0.0098 0.0099

OursCVPT-wgtGDPT-L1

0.0088 0.0070

Bathroom scene – 256 spp

• Irradiance caching

LEVERAGING COHERENCE: PREVIOUS WORK

54

A ray tracing solution for diffuse interreflectionWard et al., SIGGRAPH 1988

ImagebyWojciech Jarosz

• Irradiance caching• Photon mapping

LEVERAGING COHERENCE: PREVIOUS WORK

55

Global illumination using photon mapsJensen, EGSR 1996

ImagebyHenrik Wann Jensen

• Irradiance caching• Photon mapping• Image-space denoising

LEVERAGING COHERENCE: PREVIOUS WORK

56

ScenebyGuillermoM.LeanLlagunoImagebyBenedikt Bitterli

Energy-preserving non-linear filtersRushmeier and Ward, SIGGRAPH 1994