VR & AR path to the future an Embedded visiondsc2019.org/Docs/DSC19_TJoubert.pdf · VR & AR path to the future an Embedded vision Thierry JOUBERT ... Kilo bytes 1 024 102 400 536

www.theoris.fr©THEORIS 2019 www.theoris.fr

VR & AR path to the future

an Embedded vision

Thierry JOUBERTDSC 2019


[email protected]

• Engineer Ecole Centrale de Nantes

• 1986 - 1989 TTS – VISA4

• 1990 – 1998 Renault - SCANeR

• 2002 Co-founder

• 2011 Co-authored

• 2012 - 2015 FUI – VaRI3


Immersive systems challenges

• Re-create real sensations with a

machine

Computers help…

Real time is required

Dedicated circuits for vision, haptics

Price is constrained

• Innovation is guided by computer

market


Computer Market


1980300 000€

199030 000€

20003 000€

2010300€

201530€

20183€

Downsizingis a rule

http://static.pcinpact.com/images/bd/news/108019-intel-smartphone-atom.png



Annual sales

Growing is another rule

1980 1990 2000 2010 20201970

350 M

2 B

800 K

10 K1E+4

1E+5

1E+7

1E+9

1E+11

35 B




PC = constant price

1000€

1000€

Max Frequency

= 4GHz

More for the same price!

SSD

everywhere

Android sales

Q4-2016

PC annual sales


IoT = decreasing prices

Source: ARMAverage Selling Price of a Semiconductor Chip

>$25$10-15$3-6$1-2

Volu

me

More for less money!


• Hardware industry is mature

High competition

Consolidation

Standardization

• Worldwide market

• New business models (cloud)

Origins of the market shift…


Hardware


Alan Kay

“People who are really serious about software should make their own hardware.”


From MPU to SOC / MCU

Single chip

computer


The hardware landscape


Affordable hardware


Software


Growing compexity

Kernel

Application

Kernel

File System

Network

Application

Kernel

File System

Network

Scheduler

MMU

2D bitmap

Application

Kernel

File System

Network

Scheduler

MMU

2D bitmap

Framework

Web

3D vector

Application


Software landscape

Giga bytes1 073 741 824

Kilo bytes1 024

102 400

536 870 912

MicroPython

RAM

500 Ko Soft + Resources

50 Mo

Mega bytes1 048 576


Software coverage

RTOS

MicroEJ

Serv

ices

PlatformCortexM CortexMPCortex A

LinuxTizen Qt

Java

Intel

Apple iOS COCOA SWIFT

Android Java

C/C++

Sensor

TCP/IP

GUI

Mobile

PC

Tablet

IoT network

Windows10 .NET C#

C/C++

µPython

Display

Gateway

ESP


High end graphics software

• Thanks to the game market!

3D engines

Developer tools

Broad hardware spectrum


Choices for an immersive experience


• 3D database

• Sensors

• CPU / PPU

• GPU

• Audio

• Actuators

• Integration

• Real time

Immersion Challenges

Requirements:

• Realistic environment

• User position / actions

• World animation

• Eyes feedback

• Sound feedback

• Body feedback

• Man in the loop!!

Solutions:

Embedded


VR & AR challenges

Virtual Reality

• Immersion quality

Image quality

Image Frequency

Field of view

Resolution

• Body feedback

Hard Real Time

Infrastructure

Augmented Reality

• Real world matching

Image quality

Image Frequency

Environment tracking

• Eye tracking

Visual comfort


Cloud

• It is now part of the business model…

Saas

• Unique services for VR & AR

Big data

3D visual databases

Terrain information (OpenstreetMap, …)

Artificial Intelligence

Collaboration


Cloud + Embedded = Edge

• Next generation VR & AR

Cloud services

Local processing (Edge)

• New hardware architectures

SOC

MCU


Hololens example

Hololens 1

• HPU Tensilica

• CPU/GPU Intel

• System API

• HPU… Gen 2 ?

• CPU/GPU Qualcomm

• Simplified system API

• Cloud integration

Hololens 2


VR/AR Cloud services

Image 1

Image 2

Image 3

Feedback

Supervision

Scenario

Trafic

?


VR Streaming & Latency

• Latency ~ Serialization!!...


Lantency example


Streaming ~ Infrastructure

• Eliminate Hops

Connect to the Datacenters

Private clouds

Dedicated hardware

• Optimize data volume

Streaming compression techniques

Distribute the rendering pipeline


Be ready!

Documents

VR & AR path to the future an Embedded visiondsc2019.org/Docs/DSC19_TJoubert.pdf · VR & AR path to the future an Embedded vision Thierry JOUBERT ... Kilo bytes 1 024 102 400 536