CS291A: Real-Time High Quality RenderingSpring 2020, Lingqi Yan, UC Santa Barbara

Machine Learning for Real-Time RenderingLecture 11:

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Announcements• Good news

- Final project is due Jun 15 rather than the last week

- Same requirement as all other projects

• Bad news

- Final project showcase will be cancelled accordingly

- Will think about arrangements for the last lecture

2

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Today• Machine learning for real-time rendering

- Not just deep learning, we’ll see

- But we’ll also introduce successful applications to offline rendering

• We won’t talk about pure image processing approaches

- e.g. pix2pix, styleGAN, etc.

- These are generally considered in the Computer Vision domain

- And they are not likely to run in real-time

- And they will probably fail

3

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Machine Learning

• Deep learning is just one specific ML model

- It uses deep neural networks, but other methods still exist

- Support vector machine (SVM)

- Kernel methods, spectral methods

- MCMC, Variational Inference

- …

4

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Gaussian Mixture Models• Recall the Gaussian Mixture Models (GMM)

- In PRT, used in fitting env lights and BRDFs using SGs

- The Expectation-Maximization (EM) algorithm is indeed part of ML

5

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Difficulties applying Machine Learning

• Real-Time Computer Graphics has strict constraints on

- Performance: 10 ms is considered PROHIBITIVELY expensive

- Consider the inference time of a deep NN

- Quality: 99% correct pixels is considered a complete FAILURE

- Must always work. No cherry picking allowed

6

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Difficulties applying Machine Learning

• Specifical problems in the rendering community

- Lacks toolchains

- No “import deepshading as ds”

- Lack data

- Less than 100 high quality 3D scenes online are directly available for rendering (SUNCG, really?)

- Has a steep learning curve

- Think how difficult CS180 is

- Heavily dependent on programming skills

- Afraid of C++? Welcome to GLSL and CUDA interop

7

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Successful Machine Learning Applications

• In real-time rendering

- Compressing PRT data using a neural network

- Denoising with an autoencoder

- Deep shading

- Rendering animal fur as cloud

- BTF compression

8

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Successful Machine Learning Applications

• In offline rendering

- Deep scattering

- Learned BSSRDF

- Production denoising in animations

- NeRF to learn a light field from a single image

9

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Individual Applications

10

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Some Thoughts• Use ML to guess with guidance

• Use ML to compress data

• Use ML to speed up heavy computation

• Use ML to data-drive unknown physical rules

• Be careful to use ML directly on the output image for real-time rendering

11

CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara

Thank You!

12