Open Standards for Building Virtual and Augmented Realities

Neil Trevett | Khronos PresidentNVIDIA | VP Developer Ecosystems

2

Khronos Mission

Software

Silicon

Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for

3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing

Asian Members

3

Why Open Standards?

Bad Standards - Overprescribes implementation details

- Forces everyone to implement a lowest common denominator

- Stifles innovation-> Commoditization

Good Standards- Prescribes only interoperability

- Enables implementationdiversity

– Encourages innovation-> Differentiation

An INTEROPERABILITY STANDARD enables two interoperating entities to COMMUNICATE

A Truly OPEN standardIs not controlled by a single company – but by the whole industry

Is freely available to use by any company without royalty paymentsHas well defined IP Framework to protect standard AND member’s IP

E.g. Software <-> Hardware

Standards Grow Markets…By reducing consumer confusion and increasing capabilities and usability

… Reduce Costs …By sharing development between many

companies and driving volume

… Accelerate Time to MarketWith well-proven testing and interoperability

4

Khronos Standards for AR and VR

Vision and sensor processing -including neural network

inferencing for machine learning

High-performance, low-latency 3D Graphics

Portable interaction with VR/AR sensor, haptic and

display devices

Download 3D object scene data

5

Vulkan – New Generation 3D Graphics

Modern architecture | Low overhead| Multi-thread friendlyEXPLICIT GPU access for EFFICIENT, LOW-LATENCY, PREDICTABLE performance

Non-proprietary, royalty-free open standard ‘By the industry for the industry’Portable across multiple platforms – both desktop and mobile

VR-Specific FeaturesNOW

Multi-GPU supportMultiview Rendering

Context priorityFront buffer rendering

Evolving 3D API features forVR displays and compositors

IN DISCUSSIONVariable Rate Rendering

Tiled rendering (beam racing)

6

XR = AR + VR

V1.0 - focused on VR

V

AAfter 1.0 – equal focus on AR

7

OpenXR – Solving VR/AR Fragmentation

Proprietary Engine

VR App

1

AR App

2

ARApp

4

VRApp

3

VR Device

1

VR Device

2

VR Device

3

VR Device

5

VR Device

4

After OpenXR Wide interoperability of XR apps and devices

VRApp

1

ARApp

2

ARApp

4

VRApp

3

Proprietary Engine

VR Device

1

VR Device

3

VR Device

5

VR Device

2

VR Device

4

Before OpenXRXR Market Fragmentation

Device Integration Layer

Application Interface

WebXR WebXR

8

OpenXR Architecture

Portable AR/VRInput Device Discovery

Multiple Sensor TrackingDevice Events

Pose NormalizationHaptics Control

Optical Corrections

9

Companies Publicly Supporting OpenXR

OpenXR is a collaborative designIntegrating many lessons from proprietary ‘first-generation’ API designs

10

Input and Haptics Abstraction• Apps use abstracted Input Actions

- E.g. “Move,” “Jump,” “Teleport”• Device Events mapped to Input Actions

- For the specific system in use• Many advantages

- Content can use new devices with no code changes

- Can mix-and-match multiple input sources to create a unified UI

- Easy optional feature support (e.g. eye and body tracking)

- Future-proofing for innovation in input devices and form factors

11

OpenXR Viewport Configuration Flexibility• Applications can:

- Query for runtime supported Viewport Configurations- Applications can then set the Viewport Configurations that they plan to use- Select and change their active configuration over the lifetime of the session

Photo Credit: Dave Pape

Camera Passthrough AR Stereoscopic VR / AR Projection CAVE-like

One Viewport Two Viewports (one per eye) Twelve Viewports (six per eye)

/viewport_configuration/ar_mono/magic_window /viewport_configuration/vr/hmd /viewport_configuration/vr_cube/cave_vr

12

Feedback

OpenXR Development ProcessCall for Participation / Exploratory Group Formation

Fall F2F, October 2016: Korea

Statement of Work / Working Group FormationWinter F2F, January 2017: Vancouver

Specification WorkSpring F2F, April 2017: AmsterdamInterim F2F, July 2017: Washington

Defining the MVPFall F2F, September 2017: Chicago

Resolving Implementation IssuesWinterim F2F, November 2017: Washington

Winter F2F, January 2018: Taipei

First Public InformationGDC, March 2018

Release Provisional Specification

Ratify and release Final Specification and Enable Conformant Implementations to Ship

Present DayComing Soon

Multiple implementations Underway!Final specification will incorporate

implementation feedback

Much more detailed specification overview and SIGGRAPH session videos:

https://www.khronos.org/developers/library/2018-siggraph

First Public Demonstrations!SIGGRAPH, August 2018

Conformance Tests and Adopters Program Finalize Implementations

13

Epic ‘Showdown’ VR Demo at SIGGRAPH

Demo runs portably across StarVR and Microsoft Windows Mixed Reality headsets through the OpenXR APIs via an

Unreal Engine 4 plugin

14

Bringing VR and AR to the WebWeb XR Apps

WebXR

Native XR Apps

System-exposed AR Capabilities

3D Engines 3D Engines

Close ongoing collaboration between

WebXR and OpenXR

Khronos providing the foundation for 3D and XR

in the Web and native applications

Future versions of OpenXR will include

cross-platform extended AR functionality

15

glTF – The JPEG of 3D!

Efficient, reliable transmissionBring 3D assets into 1000s of

apps and engines

glTF 2.0 – June 2017Native AND Web APIsPhysically Based Rendering

Metallic-Roughness and Specular-Glossiness

glTF 1.0 – December 2015Primarily for WebGL

Uses GLSL for materials

glTF spec development on open GitHub – get involved!

https://github.com/KhronosGroup/glTF

Compact to TransmitSimple and Fast to Load

Describes Full ScenesRuntime Neutral

Open and Extensible

16

glTF Ecosystem

Apps and Engines

Creation Tools

Repositories

Sony 3D Creator

Oculus

Paint 3D

Collada2gltf FBX2glTFgltf-vscode

glTF-validator glTF-asset-generator

Experience

Discover

DriveDemand

Create Windows Mixed Reality Home

Users

Modo

Mixed Reality Viewer3D Builder

Prep for 3D printing

17

glTF Roadmap• glTF manages its roadmap very carefully – complexity is the enemy

- Mission #1: ensure widespread, consistent, reliable usage• Rollout new functionality first as extensions: e.g. Draco Mesh Compression

- Open source encoders and decoders are available- Extension is now shipping in more and more tools and engines

• New extensions are being developed…- E.g. Universal Compressed Texture Transmission Format

Mesh Compression Ratios

Stable CoreSpec

Domain-specificextensions stay as

extensions

New widely needed functionality ships first

as extensions

Integrate extensions into new core spec only when:1) Widespread need is confirmed by the industry

2) Widespread reliable implementation is enabled (e.g. open source)

18

Summary• Khronos is creating cutting-edge royalty-free open standards for AR/VR

- For both native and Web applications• Please join Khronos!

- We welcome members from China and Asia- Influence the direction of industry standards- Get early access to draft specifications

• More Information- www.khronos.org- ntrevett@nvidia.com- @neilt3d

WeChat: neiltrevett