Open Standards for AR - Khronos Group€¦ · Single thread per context GPU ThinDriver Explicit GPU Control Application Memory allocation Thread management Synchronization Multi-threaded

© Copyright Khronos Group 2017 - Page 1

Open Standards for ARNeil Trevett | Khronos PresidentNVIDIA VP Developer [email protected] | @neilt3d

LA, May 2017


Khronos Mission

Software

Silicon

Khronos is an International Industry Consortium of over 100 companies creating royalty-free, open standard APIs to enable software to access hardware acceleration for

3D graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing


Khronos Standards Ecosystem

VR, Vision, Neural Networks- VR/AR system portability- Tracking and odometry- Scene analysis/understanding- Neural Network inferencing

Real-time 2D/3D- Cross-platform gaming and UI

- VR and AR Displays- CAD and Product Design

- Safety-critical displays

3D for the Web- Real-time apps and games in-browser - Efficiently delivering runtime 3D assets

Parallel Computation- Machine Learning acceleration

- Embedded vision processing- High Performance Computing (HPC)


Why AR Needs Standard Acceleration APIs

PlatformFragmentation

Silicon Acceleration

Without API Standards With API Standards

ApplicationPortability

Everything runs on CPU

Standard Acceleration APIs provide PERFORMANCE, POWER AND PORTABILITY


AR Processing Flow

Vision sensor(s)

Tracking and Positioning

Geometric scene reconstruction

Semantic scene understanding

(Neural Networks)

Generate Low Latency 3D Augmentations for

display by optical system

Interact with sensor, haptic and display devices

Download 3D augmentation object

and scene data

AR Application


Vulkan – Portable New Generation GPU API

Cross Platform

7

All Major GPU Companies shipping Vulkan Drivers – for Desktop and Mobile Platforms

Mobile, Embedded and Console Platforms Supporting Vulkan

Embedded LinuxAndroid TVAndroid 7.0 Nintendo Switch

Vulkan support in Unity 5.6

Vulkan support in V5.4Late May 2017

DOTA 2 on Vulkan port of Source 2

DOOM shipping on Vulkan port of id Tech 6

Initial Vulkan mobile renderer in 4.12

Vulkan support since V1.8


Vulkan Explicit GPU Control

GPU

High-level Driver Abstraction

Layered GPU ControlContext management

Memory allocationFull GLSL compiler

Error detection

ApplicationSingle thread per context

GPU

Thin DriverExplicit GPU Control

ApplicationMemory allocation

Thread managementSynchronization

Multi-threaded generation of command buffers

Multiple Front-end Compilers

GLSL, HLSL etc.

Loadable debug and validation layers

Vulkan 1.0 provides access to OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality

but with increased performance and flexibility

SPIR-V pre-compiled shaders

Complex drivers cause overhead and inconsistent behavior across

vendors

Always active error handling

Full GLSL preprocessor and

compiler in driver

OpenGL vs. OpenGL ES

Resource management offloaded to app:

low-overhead, low-latency driver

Consistent behavior:no ‘fighting with driver

heuristics’

Validation and debug layers loaded only when needed

SPIR-V intermediate language: shading language

flexibility

Multi-threaded command creation. Multiple graphics, command and DMA queues

Unified API across all platforms with feature set

flexibility

Vulkan = high performance and low latency 3D – Ideal

for VR/AR applications


Khronos 3D APIs already powering VR rendering

The standard for desktop VR today

On millions of mobile VR devices

Powering WebVR in browsers

The future of explicit, low-latency VR

Khronos APIs for Virtual Reality

But What About INPUT Hardware for AR/VR..

Device discoveryMultiple sensor tracking

Device EventsHaptics

Parameters for optics corrections etc. etc…

OpenXR!Cross-Platform, portable AR/VR


OpenXR – Solving AR/VR Fragmentation

Proprietary Engine

VR App

1

VR App

2

VRApp

4

VRApp

3

VR Device

1

VR Device

2

VR Device

3

VR Device

5

VR Device

4

After OpenXR Wide interoperabilityof

VR apps and devices

VR App

1

VR App

2

VRApp

4

VRApp

3

Proprietary Engine

VR Device

1

VR Device

3

VR Device

5

VR Device

2

VR Device

4

Before OpenXRVR Market

Fragmentation

Device Layer

Application Interface


OpenXR Working Group Members

Design work started in December 2016Typically 12-18 months to develop a V1.0 specification


OpenXR and VR Run-times – a Win-Win

VR Run-time

Application Interface

ProprietaryAPIs

Extensions

Device Layer

ProprietaryDriver

InterfacesExtensions

Access to any OpenXR Application

Access to any OpenXR Device

Any successful standard encourages and enables healthy industry competition

OpenXR will not replace VR run-times – or outlaw existing

interfaces

OpenXR will simply provide cross-vendor APIs that can be exposed by a runtime to

access more apps and devices

OpenXR for portable AR/VR apps and devices –with initial focus on VR


OpenVX Po

wer

Eff

icie

ncy

Computation Flexibility

Dedicated Hardware

GPUCompute

Multi-coreCPUX1

X10

X100

Vision DSPs

Wide range of vision hardware architectures OpenVX provides a high-level Graph-based abstraction

->Graph-level optimizations

Can be implemented on almost any hardware or processor->

Portable, Efficient Vision Processing!

VisionNode

VisionNode

VisionNodeVision

Node

Vision Processing Graph

GPU

Vision Engines

Middleware

Applications

DSPHardware

Software Portability


OpenVX - Graph-Level Abstraction • OpenVX developers express a graph of image operations (‘Nodes’)- Using a C API

• Nodes can be executed on any hardware or processor coded in any language- Implementers can optimize under the high-level graph abstraction

• Graphs are the key to run-time power and performance optimizations- E.g. Node fusion, tiled graph processing for cache efficiency etc.

Array of Keypoints

YUVFrame

GrayFrame

CameraInput

RenderingOutput

Pyrt

Color Conversion

Channel Extract

Optical Flow

Harris Track

Image Pyramid

RGBFrame

Array of Features

Ftrt-1OpenVX Graph

OpenVX Nodes

Feature Extraction Example Graph


OpenVX Ecosystem

Multiple Shipping Implementations

Tools

E.g. AMD Open source OpenVX

Tools- Highly optimized for x86 CPU and OpenCL for GPU- “Graph Optimizer” looks

at entire processing pipeline and removes,

replaces, merges functions to improve

performance and bandwidth

- Scripting for rapid prototyping, without re-compiling, at production

performance levelshttp://gpuopen.com/compute-

product/amd-openvx/Evolving

Functionality

ExtensionsNeural Network Acceleration

Graph Save and Restore16-bit image operation

New Functionality Under Discussion

NNEF Import

Programmable user kernels with

accelerator offload


OpenVX 1.2 and Neural Net Extension• Convolution Neural Network topologies can be represented as OpenVX graphs- Layers are represented as OpenVX nodes- Layers connected by multi-dimensional tensors objects- Layer types include convolution, activation, pooling, fully-connected, soft-max- CNN nodes can be mixed with traditional vision nodes

• Import/Export Extension- Efficient handling of network Weights/Biases or complete networks

• OpenVX will be able to import NNEF files into OpenVX Neural Nets

VisionNode

VisionNode

VisionNode

Downstream ApplicationProcessing

NativeCamera Control CNN Nodes

An OpenVX graph mixing CNN nodes with traditional vision nodes


Safety Critical APIs

New Generation APIs for safety certifiable vision, graphics and

computee.g. ISO 26262 and DO-178B/C

OpenGL ES 1.0 - 2003Fixed function graphics

OpenGL ES 2.0 - 2007Shader programmable pipeline

OpenGL SC 1.0 - 2005Fixed function graphics subset

OpenGL SC 2.0 - April 2016Shader programmable pipeline subset

Experience and Guidelines

Vulkan SC being discussedSmall driver size

Advanced functionalityGraphics and compute

OpenVX SC 1.1 Released 1st May 2017Restricted “deployment” implementation

executes on the target hardware by reading the binary format and executing the pre-

compiled graphs

Khronos SCAP ‘Safety Critical Advisory Panel’Guidelines for designing APIs that

ease system certification.Open to Khronos member AND

industry experts. If interested to join contact [email protected]


glTF – Cross-Platform 3D Assets

glTF – OpenGL Transmission FormatEfficient transmission of 3D assets

Laugh Engine running on Vulkanhttps://github.com/jian-ru/laugh_engine

Now glTF 2.0 has PBR!Cool, portable materials

Rendering API independenceComing to Microsoft Office!

glTF enables portable, efficient, sophisticated 3D

assets for all 3D applications


Please Get Involved!• Information on all these standards at Khronos- www.khronos.org

• Khronos is driving to new levels of community engagement- Specifications and resources increasingly open for industry input and feedback

• Any company or organization is welcome to join Khronos- For a voice and a vote in any of these standards

• If joining is not possible – ask about a Khronos Advisory Panel- Free of charge – enables design reviews, requirements and contributions

• Still no cross-vendor camera APIs?- Is the time right for this to be a target for standardization?

• Neil Trevett- [email protected] @neilt3d

Documents

Open Standards for AR - Khronos Group€¦ · Single thread per context GPU ThinDriver Explicit GPU Control Application Memory allocation Thread management Synchronization Multi-threaded