"How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD

Copyright © 2016 AMD 1

How Computer Vision is Accelerating

the Future of Virtual Reality Allen Rush

May 2, 2016


• Virtual Reality — current view

• Computer Vision support in VR

• Future of VR with Advanced CV

• Challenges — Technical, Business

Agenda


VR Has Come a Long way

• Experience primarily thru close

display and optics

• Immersion thru control of local view

& positional info

• Factors: resolution, optics

accommodation, vergence, frame

rate, dynamic range…

• 2016: ~2M users, 200,000 developers

• >260 startups + key players


VR Experiences — Imagination to Immersion

Training and Simulation

Gaming

Big Data Visualization

Entertainment Virtual Social World Remote Presence

Medicine Education

Photorealism

in VR

Full Presence

Progression of REAL-TIME Visual Experiences

Basic 2D

Rendering

Physically

Based

Rendering

Basic 3D

Rendering

Immersive

2D Displays

Shaders VR

Level of

Presence

Today


VR + Camera, CV Basic Functions

Viewport Display vs Human FOV


Functions and processes to produce VR:

Latency challenges


• Elementary: Feedback for eye tracking, positional information

(X,Y,Z, pitch, roll, yaw, x,y,z motion- 9 DOF), extremity detection

(hands, feet, etc.)

• Fundamental: IR tracking, positioning, calibration

• Complex: deep learning training for user’s eyes expressions,

motion prediction

• Sensory fusion: combining visual, tactile, accelerometers to

model dynamic environment

Computer Vision in VR


• “Rendering of a virtual human that can purposefully interact with a real person — for

example, through speech recognition, the generation of meaningful sentences, facial

expression, emotion, skin color and tone, and muscle and joint movements — is still beyond

the capabilities of real-time computer graphics and artificial intelligence,” write

neuroscientist Maria V. Sanchez-Vives and computer scientist Mel Slater

• Key improvements in: latency, resolution, FOV, color, dynamic range, focus management

• Key additions: eye tracking, surround vision capture, position accuracy, object detection and

tracking, 3D view angle

• Key Assumption: Speeds and feeds will improve, optics will improve, additions such as eye

tracking and immersive capture will develop as natural components of VR platform

• Emergence of VR/CV development community — SDKs, app development, user groups,

metrics, reviewers, etc.

Virtual Reality — Quickly evolving and

improving, but has a way to go…


Example: Leap Motion “Virtual Hands”

• Start of Visual Projection + Visual Feedback

• Next Step: Complex rendering, recognition, depth, tracking, etc.


• Application developers: rich collection of topics to choose from

• Military (long history), medical, games (obvious), cinema (new), social media,

education, immersive work environment

• Content creation: CGI, cameras, video composite

VR Application Space — Diverse and Innovative

VR Touch UI HTC Vive Prototype Demo VR Hockey


• “The seat at the 50-yard line in the first or second

row, that might go for around $1,000 a ticket, but

what you could also do is sell that same seat for

$10 a ticket to 100,000 people.”

– Max Cohen, Oculus

• Seeing vs. experiencing: immersive and sensory

• Collaborative: Immersive telepresence

• CV: eye tracking, face reconstruction,

expression projection, etc.

Advanced VR Applications: Extending Basic

Ideas to Make Compelling Experiences

Virtual conferencing


What Does Advanced VR Look Like?

Gaze detection, learning and prediction

Automotive safety: obstacles,

cues, dictionary detection

Once baseline VR is acceptable,

many new apps, features, ideas,

and functions can be developed


• New solutions to old ideas

• Low latency rendering: creative

management of pipeline rendering

(ex: LiquidVR)

• Foveated rendering

• Sync of sensor, detection, overlay

• New Tools

• Stitching, mixing content,

workflow management

What Does it Take?

Videostitch panoramic workflow

Mix in: CV detection, recognition,

learning, tracking and tagging

VR Vision

To achieve the vision of

full presence you need

Scalable

CPUs, GPUs,

Accelerators

Sound Other Senses

Sight

Sensory

Integration


Business Opportunities and Ecosystem


• Content Developers

• Obvious: game developers. 3D games natural transition to 3D VR

• VR cinema. 360o capture, reformat, render, projection to VR display

• UI: mouse/keyboard replacement

• Captured content mixed with graphics content

• CV overlays: detection, object tagging, tracking

• Platform SDKs: standard methods for development

• Testing, metrics, distribution, support

• Optimization: compilers, libraries, porting to new platforms

Ecosystem


• Game titles: sell vs. rent vs. service?

• Collaborative: time on station? Server access

• 1+1+1=4 (cardboard + smartphone + camera = VR/CV)

• Embedded: system integration + services

Business Models


Challenges Going Forward


• Improve render and display experience

• Limit of human vision, based on foveal

cone spacing

• 2x210degx150@ 1pix/arc-min-> ~226Mpix

• At 90FPS ->~20Gpix/sec rendering speed:

settle for much less

• Need good optics, lens shading, CA, etc.

• Render complexity

• 1k-10k+(!) Ops/Pix ->100’s to1000’s GOPs

on GPU

Technical and Design Challenges —

Display and Graphics


Technical Challenges — Camera and CV

• Example: Hand and Eye Tracking

• Detail capture, frame rate,

tracking, position, prediction

• New research

• Deep learning to customize

and train


• Content Development: titles, stories, 360/180 surround, volume audio…

• Distribution: Gb of content- how to efficiently transport

• How to test, evaluate, use feedback in content creation

• Who can make money?

Challenges — Ecosystem and Business

Environment


• VR — picking up as a business!

• Improvements in historical limitations

• Cost effective solutions

• Improvements in performance

• CV — accelerates emergence of wide variety of applications

• Common attributes — detection, tracking

• Unique attributes — accuracy, categories (medical vs. games, etc.)

• Ecosystem — SDK, user groups, content creators, reviewers,

• Business — new paradigms in entertainment, efficiency, capabilities

Summary


• Alger, Mike, “Visual Design Methods for Virtual Reality” Sept 2015

• MacVey, Matthew “Exploring Future Reality”, NYC Media Lab, 2016

References

Technology

"How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD