Embed Size (px)
Citation preview
Dr. Morgan McGuire | NVIDIA Research
THE VIRTUAL FRONTIER:COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY
3
NVIDIA RESEARCH120 World-Class Ph.D. Researchers
4
VISION
Today, everyone is a high-performance computer user, with
GPUs in phones, tablets, desktops, game consoles, and cars
1977 HPC 2017 HPC1997 HPC
5
Power User Technology Pervasive
FUTURE VR
VISION
6
1. Virtual reality will be the new interface to computing for everyone
2. Virtual reality requires a new graphics systemsensors, algorithms, physics, rendering, AI, data structures, processors, optics, and displays
7
MODERN GRAPHICS SYSTEMS
8
VISUAL FIDELITY OF FILM CGI
Deadpool (Marvel)
9
VISUAL FIDELITY OF FILM CGI
Deadpool (Marvel)
10
FILM CGI: CONCEPT TO PHOTONS
Path TracerScreen24 Hz9 Mpix
Composite &Color Grade
(Hours)
ScriptCharactersCostumes
Sets
Direction
Renderer DisplayPreproduction
PerformanceModelingRigging
AnimationTexturingLighting
Simulation
Production
ParticlesTriangles
Sub-D
Primitives
12
ProductionPreproduction Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
PerformanceModelingRigging
AnimationTexturingLighting
Simulation
ScriptCharactersCostumes
Sets
RasterizationShadowMaps
AO ShadePost FX*
User Input
Direction
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
Primitives
ParticlesTriangles
HDMI, Sync
AINetwork
Simulation
13
3D GAME SYSTEM
Display30Hz2 MPix
Direction
PerformanceModelingRigging
AnimationTexturingLighting
Simulation
ScriptCharactersCostumes
Sets
User Input
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
ProductionPreproduction Renderer Display
ParticlesTriangles
Primitives
RasterizationShadowMaps
AO Shade PostFX*HDMI, Sync
AINetwork
Simulation
14
Primitives Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
15
Primitives Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
16Star Wars: Battlefront II (DICE)
1717 Forza Motorsport 6 (Turn 10 Studios)
18
7X THROUGHPUT INCREASE
3D GAME = 60 MPIX/S(1920 X 1080 @ MIN 30 FPS)
MODERN VR = 450 MPIX/S(3024 X 1680* @ MIN 90 FPS)
1920
1080
1512
1680
1512
* VR render resolution
19
Primitives Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
20
Display90 Hz5 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
3D GAME SYSTEM
Primitives Renderer Display
21
Primitives Renderer
Head Tracking
HMD
ParticlesTriangles Rasterization
ShadowMaps
Time Warp
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
MODERN VR SYSTEM
Display90 Hz5 MPix
22
LENS DISTORTION
User’s ViewPredistorted Image Optics
23
Renderer
Head Tracking
HMD
ParticlesTriangles Rasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
MODERN VR SYSTEM
Display90Hz
Primitives
24
1. Virtual reality will be the new interface to computing for everyone
2. Virtual reality requires a new graphics systemsensors, algorithms, data structures, processors, and displays
3. Pascal architecture upgrades the gaming system to modern VRwarping, lens matched shading, multiprojection, stereo projection, variable resolution
25
FUTURE GRAPHICS SYSTEMS
The remainder of the talk describes active research,including new results not previously presented in public.
These are not products.
26
LIMITS OF HUMAN PERCEPTION
x (120 pixels/degree)2
≈ 400,000,000 pixels= 200 x 1080p TVs
220o Horizontalx 135o Vertical
100,000x to 1Mx beyond modern VR
+ High dynamic range (x2), photorealistic dynamic lighting (x10,000), …
x 240 Hz
Modern VR = 450 Mpix/s
Future VR = 100,000 Mpix/s
Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html
27
Display90Hz
ParticlesTriangles
FoveatedRasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
FOVEATED RENDERING
Eye Tracking
Primitives Renderer
Head Tracking
HMD
28
FOVEATED RENDERINGConventional Approach: Aliasing
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
29
FOVEATED RENDERINGOur Approach: Perceptually Optimized
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
31
Primitives
Display90Hz
ParticlesTriangles
FoveatedRasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
BEYOND TRIANGLES
PointsText
VoxelsLight Fields
Eye Tracking
Head Tracking
Renderer HMD
32McGuire et al., Real-time global illumination with light field probes, I3D 2017
33
Renderer HMDPrimitives
Display90Hz
ParticlesTriangles
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
COMPUTATIONAL DISPLAYS
FoveatedRasterization
ShadowMaps
Time WarpPointsText
VoxelsLight Fields
VarifocalLens Distortion
Light Field
Eye Tracking
Head Tracking
34
COMPUTATIONAL DISPLAYSLight Field Display
GPU Output Display Prototype Observed Image
Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013
35
COMPUTATIONAL DISPLAYSVarifocal Optics
Hologram
Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies
Curved sunglassesLaser
36
COMPUTATIONAL DISPLAYSVarifocal Optics
Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017
37
Head Tracking
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
RendererPrimitives HMD
Display90Hz
Renderer
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
PNEUMATIC HAPTICS
FoveatedRasterization
ShadowMaps
Time WarpVarifocal
Lens Distortion
Haptics
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
Primitives
Light Field
38
Display90Hz
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
LOW LATENCYHierarchical Rendering
FoveatedRasterization
ShadowMaps
Time WarpVarifocal
Lens Distortion
Haptics
Remote GPU
Head Tracking
Light Field
Eye Tracking
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
RendererPrimitives HMD
39
LOW LATENCYHierarchical Rendering
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
Wearable TegraLocal GeForce
40
LOW LATENCYHierarchical Rendering
Wearable TegraCloud GRID Platform
Tesla GPULocal GeForce
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
41
LOW LATENCYHierarchical Rendering
Wearable Tegra
Compressed lighting data
High speed network
Cloud GRID PlatformTesla GPU
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
42
Primitives
Head Tracking
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
Renderer HMD
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
LOW LATENCYBinary Frames
FoveatedRasterization
ShadowMaps
Eye Tracking
Time Warp +Varifocal
Lens Distortion
Haptics
Remote GPU
Deep FocusDisplay
16000 Hz
Light Field
43
ULTRA LOW LATENCY & HIGH THROUGHPUTBinary Frames
Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016
44
LOW LATENCYBinary Frames
Lincoln et al., From Motion to Photons in 80 M icroseconds: Towards M inimal Latency for Virtual and Augmented Reality, IEEE VR 2016
0.08 ms
Motion Initiated
Data Received
Pixel Transmitted
Light Emitted
1
2
3
4
C1 C2 Dly
27.560µs
C1 C3 Dly
74.360µs
C1 C4 Dly
97.312µs
C2 C3 Dly
46.802µs
Ch1
Ch3
2.00 V
2.00 V
BWCh1
Ch3
2.00 VCh2
Ch4 1.00 VΏ BW
M 20.0µs Ch1 √ 1.20 V
45
LOW LATENCYOn-HMD Warping
Hardware Warping Prototype Photographed in HMD Warped Static Point Set
46
ParticlesTriangles
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
RAY & PATH TRACING
FoveatedRasterization
ShadowMaps
Eye Tracking
PointsText
VoxelsLight Fields
Time Warp +Varifocal
Lens Distortion
Haptics
Remote GPU
Deep FocusDisplay16000Hz
Denoising
Light Field
Renderer HMDPrimitives
Head Tracking
47
PATH TRACING10 rays/path
Deadpool (Marvel)
48Deadpool (Marvel)
PATH TRACING10 rays/path
Visualization of path tracing noise
1 path/pixel
49
10 rays/path1000 paths/pixel
50
DENOISING PATH TRACINGNaïve Real-time Result
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017
Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017
Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
51
DENOISING PATH TRACINGDenoised Real-Time Result
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017
Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017
Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
52
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017
Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017
Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
DENOISING PATH TRACING
53NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
AI GRAPHICS NVIDIA RESEARCH
AI Facial Animation
SIGGRAPH 2017
AI Denoising
AI Anti-Aliasing AI Light Transport
54
Display90Hz
ParticlesTriangles Rasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
MODERN VR SYSTEM
Primitives Renderer HMD
Head Tracking
55
Primitives
ParticlesTriangles
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
FUTURE VR SYSTEM
FoveatedRasterization
ShadowMaps
Eye Tracking
PointsText
VoxelsLight Fields
Time Warp +Varifocal
Lens Distortion
Haptics
Light Field
Remote GPU
Deep FocusDisplay16000Hz
Denoising
Renderer HMD
Head Tracking
56
1. Virtual reality will be the new interface to computing for everyone
2. Pascal architecture upgrades the gaming system to modern VR
GPU warping, lens matched shading, multiprojection, stereo projection, variable resolution
3. NVIDIA is innovating for a revolutionary new future VR system
computational displays, varifocal optics, foveated & cloud rendering, light fields, binary frames, on-display warping, beam racing, haptics, path tracing, denoising
57
http://research.nvidia.com
58
59
MODERN VR EXPERIENCES
Mechanics & DesignThe Climb (Crytek)SUPERHOT (Superhot Team)
Narrative & CharactersThe Labs (Valve)
Simulation & PerformanceNVIDIA VR Funhouse
ContentGoogle Earth VR
+Batman replacing Aperture
60
PROJECTOR-BASED DISPLAY
61The Vanishing of Ethan Carter (The Astronauts)
62
CHALLENGE: FOCUS CUES
Video from Narain et al., Optimal Presentation of Imagery with Focus Cueson Multi-Plane Displays, SIGGRAPH 2015
63
BEYOND TRIANGLESLight Fields
McGuire et al., Real-time global illumination with light field probes, I3D 2017
64
NVIDIA AR/VR RESEARCH
Computational Displays Light field displays and varifocal optics
Foveated Rendering Perceptually-guided rendering for massive throughput
Ultra-Low Latency Hierarchical & binary rendering, beam racing, near-display warp
Beyond Triangles Points, voxels, light fields, and text
Path Tracing Extending ray tracing leadership to cinematic quality rendering
65
NVIDIA VRWORKS SDKSIG
HT
SO
UN
D
PH
YSIC
S &
TO
UC
H
CA
PTU
RE
BRINGING REALITY TO VR
66
NVIDIA HOLODECKHANDS-ON DEMO AT SIGGRAPH
67
NVIDIA PROJECT HOLODECKPH
OTO
REA
LIS
TIC
MO
DELS
CO
LLA
BO
RATIO
N
INTERA
CTIV
E P
HYSIC
S
68
NVIDIA VRWORKS & PASCALAccelerating Modern VR
GRAPHICS HEADSET AUDIOTOUCH & PHYSICS
PROFESSIONAL
VIDEO
69
VRWORKS & PASCALHardware Acceleration for Modern VR
Single-Pass StereoLens Matched Shading & Multiprojection
Preemption for Timewarp
70
VRWORKS & PASCAL
PhysX Flex
Hairworks Flow
VRWorks Audio
Cloth
Hardware Acceleration for Modern VR
71
FOVEATED RENDERING
High-fidelity foveal
pixels
Low-fidelity peripheral
pixels
