Upload
others
View
11
Download
0
Embed Size (px)
Citation preview
© Copyright Khronos Group 2017 - Page 1
Open Standards for
3D and AR on the WebNeil Trevett | Khronos President
NVIDIA VP Developer Ecosystem
[email protected] | @neilt3d
© Copyright Khronos Group 2017 - Page 2
Topics
1 2 3
Khronos and its
API ecosystem
Latest updates
relevant to 3D and
AR on the Web
Lessons from the
past and looking
to the future
© Copyright Khronos Group 2017 - Page 3
Khronos Mission
Software
Silicon
Khronos is an International Industry Consortium of over 100 companies creating
royalty-free, open standard APIs to enable software to access hardware acceleration for
3D graphics, Virtual and Augmented Reality, Parallel Computing,
Neural Networks and Vision Processing
© Copyright Khronos Group 2017 - Page 4
Khronos Standards Ecosystem
VR, Vision, Neural Networks- VR/AR system portability
- Tracking and odometry
- Scene analysis/understanding
- Neural Network inferencing
Real-time 2D/3D- Cross-platform gaming and UI
- VR and AR Displays
- CAD and Product Design
- Safety-critical displays
3D for the Web- Real-time apps and games in-browser
- Efficiently delivering runtime 3D assets
Parallel Computation- Machine Learning acceleration
- Embedded vision processing
- High Performance Computing (HPC)
© Copyright Khronos Group 2017 - Page 5
Augmented Reality - Native Acceleration
Vision
sensor(s)
Tracking and
Positioning
Geometric scene
reconstruction
Semantic scene
understanding
(Neural Networks)
Generate Low Latency
3D Augmentations for
display by optical system
Interact with sensor, haptic
and display devices
Download 3D
augmentation object
and scene data
AR Application
How can we bring
ACCELERATED AR to
the Web?
© Copyright Khronos Group 2017 - Page 6
Compact to Transmit
Fast to Load
Describes Full Scenes
Runtime Neutral
Open and Extensible
© Copyright Khronos Group 2017 - Page 7
glTF Milestones
glTF 1.0 Spec
Ratified and
Released
Dec 2015
2012 thru 2014
Design Iteration and
Multiple
Implementations
Original motivation:
standardized way to
deliver 3D into
WebGL applications
Mar2017
glTF 2.0
Provisional spec for
industry freedback
Significant
Industry Adoption
June2017
We Are
Here!
Physically Based Rendering
for higher-quality materials
and rendering API
independence
glTF 2.0 Spec
Release
Microsoft joins
Specification Editors
Oct2016
Industry
Roadmap
Meeting
7 Months
glTF 2.0 launched today - here at Web3D!
© Copyright Khronos Group 2017 - Page 8
What’s New in glTF 2.0• Physically Based Rendering (PBR) material definitions
- Material information stored in textures
• Graphics API neutral
- Proven by engine implementations using WebGL, Vulkan and Direct3D
- GLSL materials moved to extension for existing content and specialized use cases
• Morph Targets
- Enhanced animation system
• Improvements
- Binary glTF in core
- Dozens of refinements for enhanced performance
and a tighter, clearer specification
Sketchfab User: theblueturtlehttps://sketchfab.com/models/b81008d513954189a063ff901f7abfe4
© Copyright Khronos Group 2017 - Page 9
glTF 2.0 Scene Description Structure
.gltf (JSON)
Node hierarchy, PBR material textures, cameras
.binGeometry: vertices and indices
Animation: key-frames
Skins: inverse-bind matrices
.png
.jpg
...Textures
Texture based
PBR materials
Geometry
© Copyright Khronos Group 2017 - Page 10
glTF 2.0 Scalable, Portable PBR• glTF 2.0 PBR Requirements
- Simple and inexpensive to implement - So can be everywhere – even mobile devices
- Scalable- Two combinable models – but materials continue to work even if just core supported
• Metallic-Roughness Material model
- baseColor — base color
- metallic — metalness
- roughness — roughness
- Simple to implement - mandated in core
• Specular-Glossiness Material model
- diffuse — reflected diffuse color
- specular — specular color
- glossiness — glossiness
- Slightly more resources - optional extension
Illustrations by Fraunhofer
© Copyright Khronos Group 2017 - Page 11
Industry Transitioning to glTF 2.0• Breaking changes from 1.0 to 2.0 – but processing is streamlined and simplified
- NOT significant work and great benefits by upgrading to 2.0
• Industry moving quickly to glTF 2.0 — lots of early adopters
- BabylonJS, three.js, Cesium, xeogl, instant3Dhub
• gltf-pipeline integrating glTF 2.0 updates — has glTF 1.0 to glTF 2.0 translator
- Open source - use this to support both glTF 1.0 and 2.0 or move your users to 2.0
• Converters/Translators/Validators glTF 2.0 updates nearly ready
- COLLADA2GLTF and obj2gltf translators
- Khronos Validator and Gltf-test
• Samples and Tutorials
- glTF 2.0 sample models with PBR
- Extensive glTF tutorial series in draft
Move your pipeline to glTF 2.0
Its time!
© Copyright Khronos Group 2017 - Page 12
glTF Roadmap Discussions• Incrementally ship new functionality as extensions
- For testing out new features, or for long-term optional functionality
- glTF baseline needs to remain easy to process and deploy
• Mesh Compression
- Google Draco team
• Progressive Geometry Streaming
- Fraunhofer SRC
• Basis unified compressed texture format for transmission from Binomial
- Optimized transmission format with efficient local expansion to any GPU format
• Enhanced PBR
- E.g. NVIDIA MDL
• Point Clouds
- Generated by geometry capture
• Lighting Extension
- Enhanced lighting control
• Extensions for API and language specifics
- Optional hooks for enhanced perf/functionality
- Vulkan, DX12, Metal, GLSL, HLSL, SPIR-V, Metal C++
glTF 2.0 PBR Rendering - Image courtesy instant3Dhub / instantUV - Max Limper
Roadmap Meeting at SIGGRAPH
Let me know if you would like an invite
© Copyright Khronos Group 2017 - Page 13
Khronos 3D APIs already
powering AR/VR Rendering
The standard for desktop VR today
On millions of mobile VR devices
Powering WebVR in browsers
The future of explicit, low-latency VR
Khronos APIs for AR/VR
But What About
INPUT Hardware for AR/VR..Device discovery
Multiple sensor tracking
Device Events
Haptics
Parameters for optics corrections
etc. etc…
OpenXR!Cross-Platform, portable AR/VR
© Copyright Khronos Group 2017 - Page 14
OpenXR – Solving AR/VR Fragmentation
Proprietary
Engine
VR App
1
VR App
2
VRApp
4
VRApp
3
VR Device
1
VR Device
2
VR Device
3
VR Device
5
VR Device
4
After OpenXR Wide interoperabilityof
VR apps and devices
VR App
1
VR App
2
VRApp
4
VRApp
3
Proprietary
Engine
VR Device
1
VR Device
3
VR Device
5
VR Device
2
VR Device
4
Before OpenXRVR Market
Fragmentation
Device Layer
Application Interface
© Copyright Khronos Group 2017 - Page 15
OpenXR Working Group Members
Design work started in December 2016
Typically 12-18 months to develop a V1.0 specification
© Copyright Khronos Group 2017 - Page 16
OpenXR and VR Run-times – a Win-Win
VR Run-time
Application Interface
ProprietaryAPIs
Extensions
Device Layer
ProprietaryDriver
InterfacesExtensions
Access to any
OpenXR Application
Access to any
OpenXR Device
Any successful standard encourages and
enables healthy industry competition
OpenXR will not replace
VR run-times – or outlaw existing
interfaces
OpenXR will simply provide cross-vendor
APIs that can be exposed by a runtime to
access more apps and devices
OpenXR for portable
AR/VR apps and devices –
with initial focus on VR
© Copyright Khronos Group 2017 - Page 17
API Adoption VectorsTo be relevant to the Web – a native API has to be available everywhere a Browser runs
i.e. key mobile and desktop OS
OpenGL and OpenGL ES
ships on every desktop and
mobile OS
OpenCL not adopted by
Android.
Apple switches to Metal
WebGL Next?
WebVX?
Apple not permitting Vulkan
to ship on macOS or iOS
OpenVX targeted at
embedded systems – not
desktop or mobile
© Copyright Khronos Group 2017 - Page 18
Vulkan Explicit GPU Control
GPU
High-level Driver
AbstractionLayered GPU Control
Context managementMemory allocation
Full GLSL compiler
Error detection
ApplicationSingle thread per context
GPU
Thin DriverExplicit GPU Control
ApplicationMemory allocation
Thread managementSynchronization
Multi-threaded generation of command buffers
Multiple Front-end
CompilersGLSL, HLSL etc.
Loadable debug and
validation layers
Vulkan 1.0 provides access to
OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality
but with increased performance and flexibility
SPIR-V pre-compiled shaders
Complex drivers
cause overhead
and inconsistent
behavior across
vendors
Always active
error handling
Full GLSL
preprocessor and
compiler in
driver
OpenGL vs.
OpenGL ES
Resource management
offloaded to app:
low-overhead, low-latency
driver
Consistent behavior:
no ‘fighting with driver
heuristics’
Validation and debug layers
loaded only when needed
SPIR-V intermediate
language: shading language
flexibility
Multi-threaded command
creation. Multiple graphics,
command and DMA queues
Unified API across all platforms with feature set
flexibility
Vulkan = high performance and
low latency 3D and GPU Compute
Ideal for VR/AR applications
© Copyright Khronos Group 2017 - Page 19
Vulkan – Portable New Generation GPU API
Cross Platform
7
All Major GPU Companies shipping Vulkan Drivers – for Desktop and Mobile Platforms
Mobile, Embedded and Console Platforms Supporting Vulkan
Embedded LinuxAndroid TVAndroid 7.0 Nintendo Switch
Vulkan support
in Unity 5.6
Vulkan support in V5.4
Late May 2017
DOTA 2 on Vulkan
port of Source 2
DOOM shipping on
Vulkan port of id Tech 6
Initial Vulkan mobile
renderer in 4.12
Vulkan support
since V1.8
© Copyright Khronos Group 2017 - Page 20
WebGL and WASM (Web Assembly)
C/C++ Code with
Native API calls
Emscripten
Clang/LLVM
JVM
WASM Modules
+ JS Glue Code
JavaScript
JavaScript
calls WASM
Modules
WASM modules
access Web APIs
(like WebGL)
through JS call-outs
For calling Web APIs in WASM Modules there needs to be: 1) a native API for WASM source code
2) a JS API for run-time invocation
3) a simple mapping between them to minimize JS Glue Code
JavaScript WebAPI
WebGL Next will need:1) a portable native GPU API that can run on any system
2) a JS binding of that API for use via JavaScript
3) a simple mapping between the two
© Copyright Khronos Group 2017 - Page 21
Vulkan Portability – Call for Participation
‘WebGL Next’Lift ‘Vulkan Portability’ to
JavaScript and use in
WebAssembly native code
For Web nexgen graphics
and GPU compute
API
Overlap
Analysis
Portable ‘Vulkan
Subset’ API
Specification
A Portability Solution needs to address
APIs and shading languages
Open source compilers/translators for
shading and intermediate languages
MIR
DX IL
GLSL
MSL
HLSL
Vulkan Portability SolutionC/C++ Portability API Library
+ Shading Language tools
All open source
Use Feature
Sets to remove
non-portable
functionality
Use Extensions
to add
functionality
e.g. security and
robustness for
the Web
Vulkan is non-proprietary and is
already designed to be portable
No single API
on all systems
https://github.com/KhronosGroup/WebGLNext-Proposals/tree/master/Obsidian-Mozilla
Mozilla Obisidian ‘WebGL Next’ Proposal
© Copyright Khronos Group 2017 - Page 22
2007OpenGL ES 2.0
2012OpenGL ES 3.0
32-bit integers and floats
NPOT, 3D/depth textures
Texture arrays
Multiple Render Targets
Programmable
Vertex and
fragment shaders
2016
General Purpose Compute
Multi-threaded Rendering
Low-latency renderring
2011
WebGL 1.0
March 2017
WebGL 2.0Conformance Testing is vital for
Cross-Platform ReliabilityWebGL 2.0 conformance tests are very thorough
10x more tests than WebGL 1.0 tests
‘WebGL
Next’?Pervasive OpenGL ES 2.0OpenGL and OpenGL ES ship on
every desktop and mobile OS
WebGL Evolution
Vulkan
Portability
Apple
GPUWeb W3C
Community Group?
??
WebGL 2.0 now available
in Chrome/Firefox!
Edge/Safari: plan to ship
Already up to 50% of all
browsers
2014OpenGL ES 3.1Compute Shaders
Apple not shipping ES 3.1
Compute shader extension?
© Copyright Khronos Group 2017 - Page 23
OpenVX - Portable, Efficient Vision Processing• OpenVX developers express a graph of vision and Neural Network operations (‘Nodes’)
- Using a C API
• Graphs isolate the programmer from hardware and enable significant optimizations
- Implementers can optimize – including node fusion and memory tiled caching
• Nodes can be executed on any hardware or processor coded in any language
- Including Vulkan for mobile and embedded OS
• BUT – native OpenVX is targeted at embedded – not desktop/mobile OS
- Perhaps a JavaScript Web graph API over Vulkan is still useful?
VisionNode
VisionNode
VisionNodeCNN Nodes
OpenVX graph mixing CNN nodes with traditional vision nodes
‘WebVX’??
Possible Native StackC API to Build Graph
Nodes accelerated by Vulkancompute shaders
Potential Web Stack??JS API to Build Graph
Nodes accelerated by Vulkancompute shaders
© Copyright Khronos Group 2017 - Page 24
Why AR Needs Standard Acceleration APIs
Platform
Fragmentation
Silicon
Acceleration
Without API Standards With API Standards
Application
Portability
Everything
runs on CPU
Standard Acceleration APIs provide PERFORMANCE, POWER AND PORTABILITY
© Copyright Khronos Group 2017 - Page 25
Consequences of Not Enough Acceleration
© Copyright Khronos Group 2017 - Page 26
Layered API Ecosystem Examples
Application
Middleware
Browser
Native
WebVX?
Portability
3D Web Apps VR Web Apps AR Web Apps
High-level AR APIs similar to
proprietary platforms:
Android Tango, Apple ARKit?
Use
Case
Targ
ete
d
Div
ers
e O
pen S
ourc
e P
roje
cts
Sin
gle
im
ple
menta
tions
Genera
l Purp
ose
Defi
ned s
pecif
icati
on
Mult
iple
Im
ple
menta
tions
GPU Compute for Vision
and Machine Learning +
Rendering
General purpose vision
processing/Neural Net
for the Web – enables
Web innovation
In Discussion!!
= WebXR?
© Copyright Khronos Group 2017 - Page 27
Augmented Reality - Web Acceleration
Vision
sensor(s)
Tracking and
Positioning
Geometric scene
reconstruction
Semantic scene
understanding
(Neural Networks)
Generate Low Latency
3D Augmentations for
display by optical system
Interact with sensor, haptic
and display devices
Download 3D
augmentation object
and scene data
AR Application
‘WebVX’
‘WebGL Next’
© Copyright Khronos Group 2017 - Page 28
Please Get Involved!• Information on all these standards at Khronos
- www.khronos.org
• Khronos is driving to new levels of community engagement
- Specifications and resources increasingly open for industry input and feedback
•Any company or organization is welcome to join Khronos
- For a voice and a vote in any of these standards
• If joining is not possible – ask about a Khronos Advisory Panel
- Free of charge – enables design reviews, requirements and contributions
• Khronos is always open to liaison projects with Web3D
- The industry benefits from everyone working more closely
•Neil Trevett
- @neilt3d