View
9
Download
0
Category
Preview:
Citation preview
CS291A: Real-Time High Quality RenderingSpring 2020, Lingqi Yan, UC Santa Barbara
Machine Learning for Real-Time RenderingLecture 11:
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Announcements• Good news
- Final project is due Jun 15 rather than the last week
- Same requirement as all other projects
• Bad news
- Final project showcase will be cancelled accordingly
- Will think about arrangements for the last lecture
2
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Today• Machine learning for real-time rendering
- Not just deep learning, we’ll see
- But we’ll also introduce successful applications to offline rendering
• We won’t talk about pure image processing approaches
- e.g. pix2pix, styleGAN, etc.
- These are generally considered in the Computer Vision domain
- And they are not likely to run in real-time
- And they will probably fail
3
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Machine Learning
• Deep learning is just one specific ML model
- It uses deep neural networks, but other methods still exist
- Support vector machine (SVM)
- Kernel methods, spectral methods
- MCMC, Variational Inference
- …
4
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Gaussian Mixture Models• Recall the Gaussian Mixture Models (GMM)
- In PRT, used in fitting env lights and BRDFs using SGs
- The Expectation-Maximization (EM) algorithm is indeed part of ML
5
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Difficulties applying Machine Learning
• Real-Time Computer Graphics has strict constraints on
- Performance: 10 ms is considered PROHIBITIVELY expensive
- Consider the inference time of a deep NN
- Quality: 99% correct pixels is considered a complete FAILURE
- Must always work. No cherry picking allowed
6
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Difficulties applying Machine Learning
• Specifical problems in the rendering community
- Lacks toolchains
- No “import deepshading as ds”
- Lack data
- Less than 100 high quality 3D scenes online are directly available for rendering (SUNCG, really?)
- Has a steep learning curve
- Think how difficult CS180 is
- Heavily dependent on programming skills
- Afraid of C++? Welcome to GLSL and CUDA interop
7
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Successful Machine Learning Applications
• In real-time rendering
- Compressing PRT data using a neural network
- Denoising with an autoencoder
- Deep shading
- Rendering animal fur as cloud
- BTF compression
8
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Successful Machine Learning Applications
• In offline rendering
- Deep scattering
- Learned BSSRDF
- Production denoising in animations
- NeRF to learn a light field from a single image
9
CS291A, Spring 2020 Lingqi Yan, UC Santa Barbara
Some Thoughts• Use ML to guess with guidance
• Use ML to compress data
• Use ML to speed up heavy computation
• Use ML to data-drive unknown physical rules
• Be careful to use ML directly on the output image for real-time rendering
11
Recommended