Virtual Reality Stereoscopy : A year of prototypes and conclusions

Virtual Reality Stereoscopy A year of prototypes and conclusions.

Diego Bezares. @diegobez

What if... ?

Accommodation reflex

Convergence

Stereoscopy

IPD. Interpupilar distance

- How does the brain “view” different IPD’s ?

Stereoscopy

Normal Mapping vs. Parallax MappingThe technique known as “normal mapping” provides realistic lighting cues to convey depth and texture

without adding to the vertex detail of a given 3D model. Although widely used in modern games, it is much

less compelling when viewed in stereoscopic 3D. Because normal mapping does not account for binocular

disparity or motion parallax, it produces an image akin to a flat texture painted onto the object model.

“Parallax mapping” builds on the idea of normal mapping, but accounts for depth cues normal mapping

does not. Parallax mapping shifts the texture coordinates of the sampled surface texture by using an

additional height map provided by the content creator. The texture coordinate shift is applied using the per-

pixel or per-vertex view direction calculated at the shader level. Parallax mapping is best utilized on

surfaces with fine detail that would not affect the collision surface, such as brick walls or cobblestone

pathways.

Why stereoscopy ?

Presence : The sense of “being there” .

An illusion that a mediated experience is not mediated.

Pre-render

¿ what if…. ?

Scene Prerendered to equirectangular stereo

360 panorama capture

360 panorama capture

Animated waves + Kite shadow

Virtual cameras movement

Omni Directional Stereo Content

https://developers.google.com/vr/jump/rendering-ods-content.pdf Google

Omni-directional stereo (ODS) is a projection model for stereo 360 degree videos. It’s designed for VR viewing with a head-mounted display (HMD).

ODS uses a special projection format which has the following advantages: ● It is panoramic and stereo everywhere—there are no bad seams or dead zones (except directly above and below the camera, which we will discuss later). ● It's pre-rendered—like film and video, it plays well on all devices. ● It’s encoded as two video streams—it can be stored, edited, and transmitted using conventional tools.

stereo ?

ODS

4096 x 2048 pixels -> 4096 camera positions

ODS : No manual stitching- Stitching : the process of combining multiple images with overlapping fields of view to produce a segmented panorama

Stitching errors ->

Google Jump is using machine learning for stitching

Unity3D ODS camera implementation

- Problemas en los polos

BlackJackVR

Cubemap projection vs equirectangular

20% improvement

Cubemap sizeEjemplo : GearVR.Resolución : 2560 x 1440 pixels => 1280 x 1440 cada ojoFOV : 96 gradosPíxeles por grado : 1280 / 96 = 13.4 horizontal, 1440 / 96 = 15 verticalSi giramos podemos ver 360º horizontales -> 360 x 13.4 = 4824 píxelesSi giramos podemos ver 180º verticales-> 180 x 15 = 2700 píxeles

render target bigger ( for distorsion correction ), 1.4x

Positional Tracking support

¿ What if ... ?

Pedro Fernando Gómez

@pedrofgomez

pedrofe.com

Positional Tracking support

Pedro Fernando Gómez

@pedrofgomez

pedrofe.com

lights, stereo billboard

Full screen prerendered.

Viable :High poly, complex scenes, complex lightning.

Not-realtime rendering engines , VR cameras like : Pedro Fernando Gómez VR Camera for Arnold : http://pedrofe.com/oculus-camera/

Complex, expensive.

What if... ?

Capturing reality

- Arduino + step motor- OpenCV :

video -> imagen ODS

Capturing reality

http://cctoolkit.vectorcult.com/

Capturing realityhttps://docs.google.com/document/d/14CpJQKUeeFDt_P07oStEBbJqbdJikMvgv8tIyHb8F2I/edit?usp=sharinghttp://mistralsocialgames.com/vr/test2.html

1 minute 360 degrees, 2 cameras 60fps 1080p.2 pixel columns per position.60 fps x 60 secs/min x 2 pixels = 7200 pixels

7200 x 1900 pixels per eye.

Arduino one, accelstep library, 28BYJ-48 step motor

Animated, interactive scene

- prerendered animations.- Loops : tree leaves, characters : idle -> action -> idle.- Interactivity- Integration with Realtime 3D objects.

What if ... ?

VR Video

- 360 degrees, just 100 degrees at the screen..- Big video files : resolution, 360º, stereo, etc...- Decoding processing limited : 20mbs high end mobile, 50mbs PC.- Bad stereoscopy ( 16 camera positions, stitching errors … ).- Complex stitching, expensive.

VR Video

- https://t.co/I3BKHJTSIt- https://drive.google.com/file/d/0B39SXMRp_xSTQmZGRl94ZjloUXc/view?usp=sharing

VR VideoArc of alpha, beta degrees

- stitchado bordes

VR VideoStitch

VR Video

Blend

VR Video

- Chroma- OpenCV : splitting static background and foreground.

Diego bezares. @Diegobez

Thanks !GearVR PlaystationVRCardboard SteamVR

XBOX Scorpio

Oculus RIFT

2014June

2015November

2016March

2016April

2016October

2016Fall

2017Late

Daydream

Locomotion

Resolución en R.V

Resolución en R.V

Píxels por grado de visión.

Render Target

Trends