Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.
www.esi-group.com
SAN JOSÉ | 08 – 11 MAY 2017
DELIVERING IMMERSIVE EXPERIENCESTHROUGH GPU VIRTUALIZATION AND STREAMING
Jan Wurster08 MAY 2017 | S7203 5:00pm to 5:50pm
2www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
“A pioneer and world-leading provider in Virtual Prototyping.”
— www.esi-group.com
3www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACEScourtesy of Getty Images
4www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACEScourtesy of BMW Group
5www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACES
6www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACEScourtesy of Valmet Automotive
7www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.
“Augmented And Virtual Reality Fuel The Future Workplace“
Forbes Mag, Nov 2016
8www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
Once upon a time: Physical Prototypes as part ofthe workspace
Interacting with peers
Continuous validation andpractical engineering
Established VR: Virtual Prototypes in Virtual Environments
All but removes the need for physical prototypes, saving time, effort and resources
Great for Collaboration!
Cost, complexity, maintenance and availability in digital prototypes sometimes lead to less validation andcollaboration.
9www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
Workplace VR! Finally: Affordable immersive hardware!
However .. Collaboration?
Efficient use of expensive (and noisy) resources?
Complexity of setup and maintenance?
Software affordabiblity?
10www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
11www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
12www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
• Offer the benefits of VR in engineering to non-experts
• Adopt to and support changes in infrastructure
• Integrate and support processes in the 4.0 enterprise
• Lower the technical obstacleswithout compromising quality ofresults
• Use the opportunity of mobile VR ubiquity - match today‘s flexible workstyles
SOFTWARE AFFORDABILITY
HARDWARE ACCESSIBILITY
INTUITIVE USABILITY
Virtualization& Cloud
Technology
Immersive
Virtual Reality Experience
13www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Working Environments in Engineering
High Quality Immersive Virtual Reality to the desk
VR IN THE DATACENTER
Untether Virtual Reality experiences from the workspace
IMMERSIVE CLOUD EXPERIENCES AND MOBILE VR
Collaboratively Investigate, Evaluate and Decide
COLLABORATIVE IMMERSIVE SPACES
14www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
The Engineering Workspace
OBJECTIVES & REQUIREMENTS INFRASTRUCTURE
• Accessibility & Democratization• Virtual Reality at the Workplace –
without the technical complexity ofworkstation-based VR
• High Quality ImmersiveEnvironments (CAVE, Powerwall)
• Share Resources• Exploit available computing
resources more efficiently throughsharing
• Enterprise Visualization Datacenter• GPU virtualization to support thin /
mobile clients
• High quality streaming via High Bandwidth / low latency enterprisenetworks
• Building a GPU infrastructure in Datacenters based on NVIDIA GRID / Tesla infrastructure to support bothtraditional engineering and immersivevirtual reality applications
15www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Thin Client Prototyping Challenges
HMD prototyping Scenario requires the lowest possible latency for full end to end roundtrip
at the highest feasible throughput
with the most challenging aspects of user comfort
Setup Vive - capable Workstation as a simplified backend
Dual M6000 NVIDIA GPUs
IC.IDO with Helios Renderer based on NVIDIA NV-Pro pipeline Talk: S7184: ENGINEERING DATASETS ON HMDS
Standard 1GBit Ethernet network
Standard-Issue low power Notebook as a thin client
Visit us atBooth 708
-Try the demo!
May 8-11, 2017 | Silicon Valley
16www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Thin Client Prototyping Challenges
BANDWIDTH
LATENCY
INFRASTRUCTURE INTEGRATION
USER COMFORT
17www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Headroomup to 40ms
Proc & Display 10 ms
Theory: Streaming Virtual Reality Content
HMD Scenario
HMD Scenario Throughput Target Rate 90 fps
45 fps (with device processing: warping/reprojection)
Motion to Photon Latency Imperceptable: <= 20ms
Acceptable: < 80ms
NOTE: Non-Gaming Scenario!
BANDWIDTHLATENCY
ApplicationRendering10-20 ms
Tracking5 ms
BackendRx, App,Visualize, Encode, Tx25ms
Thin ClientRx, Decode, Display20ms
Thin ClientInput & Tx8ms
Network2 ms
Network4 ms
Proc & Display 10 ms
Decode, Render5 ms
Encode3 ms
ApplicationRendering10-20 ms
Tracking5 ms
Streaming VR at a total latency below 80ms seems feasible.
18www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Thin Client – Solution Architecture
Thin
Clie
nt
HELIOS
IC.I
DO
INST
AN
CE NVIDIA CAPTURE
SDKSTREAMING
NVENC NVIFR
STREAMING SERVER
NVIDIA GRIDvGPU
VCA
TRACKING
NVIDIA Codec
SDK
OpenVR
FRA
ME
REC
EIV
ER
GPU
INFRASTRUCTUREBANDWIDTH
19www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Measurements: Streaming Virtual Reality Content
Backend: Dell T5810 Workstation 1x Quadro M6000
Thin Client: Dell Notebook NVIDIA Quadro
(Measurements averaged over several minutes of interactions)
BANDWIDTHLATENCY
Proc & Display 10 ms
NVDEC, Render8 ms
NVENC3 ms
App, Render8 ms
Tracking5 ms
Local 1GBase-T Network
Rx,Application,Rendering,
Encoding<12 ms
Tx, Rx (Frame) 10ms
Motion toPhoton Latency
~ 45ms
Latency Overhead forstreaming: <20ms
(tcp (websockets), NVENC, NVDEC)
20www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Measurements: Streaming Virtual Reality Content
Results Throughput tested up to 250 fps
(1GBase-T local network)
Trading off throughput and latency toimage quality required!
90 fps are achievable
45 fps when combined with device-side processing can appear superiordue to higher quality encoding
BANDWIDTHLATENCY
0,00
10,00
20,00
30,00
40,00
50,00
60,00
70,00
80,00
90,00
100,00
90 fps (1GBase-T) 180 fps (1GBase-T) 80 fps (802.11ac)
Latency Measurementsmin max avg
21www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Streaming Virtual Reality Content
Improving Headroom Trying to parallelize sending tracking data /
receiving frames / decoding & displaying frameson the client did not yield significantimprovements
Optimization of system latency should focus theapplication and visualization subsystems.
Avoiding and improving Network-side latency forframe data provides the biggest remainingpotential
Introduce Redundancy for stream rate stability in clogged / non-dedicated network protocols (oreven wireless)
USER COMFORT
Proc & Display 10 ms
NVDEC, Render8 ms
NVENC3 ms
App, Render8 ms
Tracking5 ms
Local 1GBase-T Network
App and Backend Image Generation
Network Protocol
Client Decoding andRendering
22www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Further Work: Streaming Virtual Reality Content
Increasing User Comfort Composite Image Generation
Stream high complexity data
Render low complexity data locally
User Representation Introduce avatar representation (*) as low-latency elements rendered locally
Optimize Streaming Protocol Parallelized UDP streams, reintegrated based on first received packet
Stream Video Quality, Resolution Optimize streaming resolution to reflect properties of HMD optics
Optimize Codec and parameter choice
USER COMFORT
(*) Augmenting immersive cinematic experience/scene with user body visualizationChen, Joshua Yang, 2016, http://hdl.handle.net/10092/12144Nasum Virtualis: A Simple Technique for Reducing Simulator SicknessDavid Matthew Whittinghill, Bradley Ziegler, James Moore, and Tristan Case
23www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
On Premise / Enterprise Cloud Stack
NVIDIA GRID | HypervisorBare Metal
Conceptualizing an Immersive Datacenter Infrastructure
vGPU vGPUvGPU
GPUEnterprise Storage
Ceph, Swift, ..
DesktopApp
IC.IDOInstance(Master)
Thin ClientImmersiveSetups
Application & Visualization
VirtualizationDatacenter
INFRASTRUCTURE
vGPU
GPU
Enterprise Network
IC.IDOInstance(Master)
IDO.Cooperate Cluster
24www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
On Premise / Enterprise Cloud Stack
NVIDIA GRID | HypervisorBare Metal
Conceptualizing an Immersive Datacenter Infrastructure
vGPU vGPUvGPUvGPU
GPUGPUEnterprise Storage
Ceph, Swift, ..
IC.IDOInstance(Master)
IC.IDOInstance(Slave)
IC.IDOInstance(Slave)
Visualization Cluster
DesktopApp
IC.IDOInstance(Master)
Enterprise Network
Thin ClientImmersiveSetups
Application & Visualization
VirtualizationDatacenter
INFRASTRUCTURE
Immersive Projection Setups
25www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
On Premise / Enterprise Cloud Stack
NVIDIA GRID | HypervisorBare Metal
Conceptualizing an Immersive Datacenter Infrastructure
vGPU vGPUvGPUvGPU
GPUGPUEnterprise Storage
Ceph, Swift, ..
IC.IDOInstance(Master)
IC.IDOInstance(Slave)
IC.IDOInstance(Slave)
Visualization Cluster
vrifyCloudInstance
IC.IDOInstance(Master)
Enterprise Network
Thin ClientImmersiveSetups
Application & Visualization
VirtualizationDatacenter
INFRASTRUCTURE
Immersive Projection Setups
IDO.Cooperate Cluster
26www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Immersive Cloud Experiences & mobile VR
IndustrialDesigners
Design Engineers
Manufacturing& Integration
Engineers
Maintenance & repair
operators
Makers & Enthusiasts
Engineering Schools
Educators
27www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Personal Immersive VR HMDs
Virtual Reality Technology Evolution
Projection-based VRCAVE, Powerwall
Co
llab
ora
tive
VR
Imm
ers
ive
VR
Mobile VRCardboard, GearVR, Daydream
mo
bile
VR
Come visitbooth 704 at
GTC 2017!
Enterprises & Institutes
Engineering & Research
Enterprises & Institutes
Recreation & Experiences
Gamers & Enthusiasts
Engineering & Research
ContentContent
Audience
AudienceEnterprises & Institutes
Gamers & Enthusiasts
„Everybody Else“
Audience
360° Videos, Games, ?Content
Established Virtual Reality SolutionsImmersive Virtual Engineering
28www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Designing for mobile Virtual Reality
SCENARIO REQUIREMENT PLATFORM CHARACTERISTICS
• VR Rendering Paradigms• Low latency
• 60++ FPS
• Highly dynamic and interactive
• Engineering Scenario Characteristics• 20++ Million Triangles
• Physics Simulation
• Advanced Appearances
• Global Illumination
• Limited Device Resources
• Stereoscopic Viewpoint and Tracking • No positional head tracking
• Screen and Lens Quality• Low resolution and optical quality
29www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Overcoming Limitations
COMPUTING POWER
GPU RESOURCES
INTERACTION CAPABILITIES
3DOF TRACKING
30www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Overcoming Limitations
Decoupled Remote Visualization
Cloud Computing Infrastructure
Hybrid User Experience
31www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Hybrid User ExperienceINTERACTION3DOF TRACKING
WEB
CLI
ENT
WEB
CLI
ENT
VR
WEB
CLI
ENT
Webbased client for Mobile VR Immersive Virtual Reality as a ‚second screen‘
Client-local User Interface - low latency and direct interaction
HTML5 - platform-agnostic , responsive UI
Flexible input and usability
WEB
SER
VIC
E
VIS
UA
LIZA
TIO
NSE
RV
ICE
Physics
Visualization
Application
AP
PLI
CA
TIO
N
SER
VIC
E
Auth
AP
PLI
CA
TIO
N
SER
VIC
E Auth
AP
PLI
CA
TIO
NSE
RV
ICE
Storage
Auth
Scaling
PUBLIC / PRIVATE CLOUD INFRASTRUCTURE
32www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Hybrid User ExperienceINTERACTION3DOF TRACKING
WEB
CLI
ENT
WEB
CLI
ENT
VR
WEB
CLI
ENT
Engineering use cases require scaling in multiple dimensions Multi-User and Multi-Tenant
Compute capacity for realtime physics
Collaboration and sharing of results
WEB
SER
VIC
E
VIS
UA
LIZA
TIO
NSE
RV
ICE
Physics
Visualization
Application
AP
PLI
CA
TIO
N
SER
VIC
E
Auth
AP
PLI
CA
TIO
N
SER
VIC
E Auth
AP
PLI
CA
TIO
NSE
RV
ICE
Storage
Auth
Scaling
PUBLIC / PRIVATE CLOUD INFRASTRUCTURE
33www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
VIS
UA
LIZA
TIO
N S
ERV
ICE
VIS
UA
LIZA
TIO
N S
ERV
ICE
VR
WEB
CLI
ENT
Decoupled Remote Visualization PipelineGPU RESOURCES 3DOF TRACKING
OPTIX
OPENGL
VIS
UA
LIZA
TIO
N S
ERV
ICE
NVIDIA CAPTURESDK
STREAMING NVENC NVIFR
STREAMING SERVER
NVIDIA GRIDvGPU
VCA
SCALABLE NVIDIA GRID vGPU INFRASTRUCTURE
34www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Rendering Stereoscopic Panoramics
Decoupled Remote Visualization
LEFT
RIGHT
Top / Bottom Format at >= 2048x2048 resolution
Omnidirectional Stereo Low visualizuation latency
Feasible bandwidth for streaming
Interactive / dynamic content
Straightforward through Raytracing
Can be approximated throughRasterization
Plenoptic Rendering / Lightfields Some location independence
Static content
Synthesis Non-Realtime
35www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ODS Image Synthesis
Rendering Stereoscopic Panoramics
STEREO CAMERA SETUP ODS APPROXIMATED ODS APPROXIMATED
ReferencesP.D. Bourke. “Synthetic Stereoscopic Panoramic Images”Google Developers – „Rendering Omni‐directional Stereo Content“
36www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Displaying Stereoscopic Panoramics
Decoupled Remote Visualization
Tracking & Projection
Warping & Lens Correction
Decoding
Immersive UI Overlay
WEB
CLI
ENT
STR
EAMWebVR Client for mobile devices
Device APIs for efficient GPU-enabledDecoding
Equirectangular imagestexturemapped to left / right eyespheres
Client-side User Interface Overlay
Device APIs for Warping and Lens Correction
Compatible to GearVR, Cardboard, Rift, Vive, Daydream
37www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
“Rendering Omni‐directional Stereo Content”— Source: Google Developers,https://developers.google.com/vr/jump/rendering-ods-content.pdf
“Synthetic Stereoscopic Panoramic Images”
— Source: P.D. Bourke. Lecture Notes in Computer Science (LNCS), Springer,
ISBN 978-3-540-46304-7, Volume 4270, 2006, pp 147-155
“Nasum Virtualis: A Simple Technique for Reducing Simulator Sickness”
— Source: David Matthew Whittinghill, Bradley Ziegler, James Moore, and Tristan Case,
http://www.purdue.edu/newsroom/releases/2015/Q1/virtual-nose-may-reduce-simulator-sickness-in-video-games.html
“Augmenting immersive cinematic experience/scene with user body visualization”
— Source: Chen, Joshua Yang, 2016, http://hdl.handle.net/10092/12144
38www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Acknowledgements and Team
‣ NVIDIA DevTech Team
‣ Don Breda• NVIDIA – ESI Alliance Manager
‣ NVIDIA Capture• https://developer.nvidia.com/capture-sdk
‣ Andreas Kleber• HTC Vive Device Support• Oculus Rift Device Support
‣ Thomas Kipshagen• Streaming Client and Protocol• Optimizations
‣ Andreas Mank• HMD integration• Raytracing Omidirectional Stereo• Thin Client Threading
‣ Michael Hofmann• Integration with Helios and Porting
‣ Trevor Dodds• Streaming Server
39www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
THANK YOU
May 8-11, 2017 | Silicon Valley
http://icido.com http://vrify.esi-group.com