76
Sébastien Kuntz [email protected] CEO The Challenges of Creating VR Applications in Unity

SAE AR/VR - The challenges of creating a VR application with Unity

Embed Size (px)

DESCRIPTION

Unity3D is a great authoring tool to create 3D applications. Regarding immersive virtual reality, its capabilities are limited. We will discuss the challenges and solutions to create great VR applications in Unity!

Citation preview

Page 1: SAE AR/VR - The challenges of creating a VR application with Unity

Sébastien [email protected]!CEO

The Challenges of!Creating VR Applications!

in Unity

Page 2: SAE AR/VR - The challenges of creating a VR application with Unity

!

• Founder & President!

• 12 years in VR!

!

• - French railways!

!

Sébastien Kuntz

Page 3: SAE AR/VR - The challenges of creating a VR application with Unity

• Simplify VR!

• Creation!

• Deployment!

• So YOU can spend more time creating applications

Page 4: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Rendering speed!

• MiddleVR for Unity!• Conclusion

Page 5: SAE AR/VR - The challenges of creating a VR application with Unity

Plan

Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 6: SAE AR/VR - The challenges of creating a VR application with Unity

• Platform for creating interactive 3d applications!

• Widely used!

• 2 million registered developers!

• Certified by the US Army and Air Force!

• Clients : Bigpoint, Cartoon Network, Coca-Cola, Disney, Electronic Arts, LEGO, Microsoft, NASA, Nickelodeon, Ubisoft, US Army, Warner Bros.

Page 7: SAE AR/VR - The challenges of creating a VR application with Unity

• High-quality rendering!

• Deferred renderer!

• Occlusion culling!

• Realtime shadows!

• Ambient occlusion!

• Light mapping

Page 8: SAE AR/VR - The challenges of creating a VR application with Unity
Page 9: SAE AR/VR - The challenges of creating a VR application with Unity

Unity 3D Features• Programming!

• C#, Javascript!

• Profiler!

• Physics!

• NVidia PhysX!

• Cloth!

• Soft/rigid bodies!

• Ragdolls!

• Cars

3D formats !

FBX, Collada, 3DS, OBJ…!

Audio engine!

Networking!

Terrain

Page 10: SAE AR/VR - The challenges of creating a VR application with Unity

Editor

• Simple to use!

• Create your own tools inside Unity

Page 11: SAE AR/VR - The challenges of creating a VR application with Unity

Asset store

• Buy / sell assets!

• 3D models!

• Textures!

• Shaders!

• Scripts!

• Editor plugins

Page 12: SAE AR/VR - The challenges of creating a VR application with Unity

Desktop

Page 13: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!

Challenges!• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 14: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 15: SAE AR/VR - The challenges of creating a VR application with Unity

Virtual reality

Presence in a virtual world

(c) eMagin(c) Barco

Page 16: SAE AR/VR - The challenges of creating a VR application with Unity

Experience of presenceis in your brain !

Page 17: SAE AR/VR - The challenges of creating a VR application with Unity

Cognitive presencePerceptive presence

Page 18: SAE AR/VR - The challenges of creating a VR application with Unity

• Build hardware

To create and maintain presence, you need to ...

• Build software

• Build user experience

Page 19: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!

Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 20: SAE AR/VR - The challenges of creating a VR application with Unity

Device management

Physical device!!

Driver Application

Page 21: SAE AR/VR - The challenges of creating a VR application with Unity

Device management

Tracker 1! Driver Tracker 1

Application

Tracker 2!!

Driver Tracker 2

Page 22: SAE AR/VR - The challenges of creating a VR application with Unity
Page 23: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!• Device management!

Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 24: SAE AR/VR - The challenges of creating a VR application with Unity

Correct perspective

(c) Johnny Lee

Page 25: SAE AR/VR - The challenges of creating a VR application with Unity

Correct perspective

Page 26: SAE AR/VR - The challenges of creating a VR application with Unity

Stereoscopy

Page 27: SAE AR/VR - The challenges of creating a VR application with Unity

Warping & Blending

Page 28: SAE AR/VR - The challenges of creating a VR application with Unity

Graphics cards limitations

• Number of outputs!

• Processing power

Page 29: SAE AR/VR - The challenges of creating a VR application with Unity

multiple graphics cards

• Pro!

• More outputs!

• More processing power!

• Con!

• 3D engines are not optimized for multi-pipe

Page 30: SAE AR/VR - The challenges of creating a VR application with Unity

Cluster

• Pro!

• More outputs!

• More processing power!

• Con!

• 3D engines don’t support clustering

Page 31: SAE AR/VR - The challenges of creating a VR application with Unity

• No seams across displays require 3 layers of sync!

• 3D objects synchro (Scenelock)!

• New images display (Swaplock)!

• Left/Right eyes (Genlock)

Cluster

Page 32: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!

Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 33: SAE AR/VR - The challenges of creating a VR application with Unity

Interactions

• Many different 3D user interactions!

• Navigation!

• Selection!

• Manipulation!

• Draw!

• System control!

• etc.

Page 34: SAE AR/VR - The challenges of creating a VR application with Unity

Interactions are dependent on hardware

Page 35: SAE AR/VR - The challenges of creating a VR application with Unity

Menus

Page 36: SAE AR/VR - The challenges of creating a VR application with Unity

Haptics (force feedback)

Page 37: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!

Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Page 38: SAE AR/VR - The challenges of creating a VR application with Unity

Deployment

• Modify application to!

• Change drivers!

• Change displays!

• Change interactions

Page 39: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!

Rendering speed!• MiddleVR for Unity!• Conclusion

Page 40: SAE AR/VR - The challenges of creating a VR application with Unity

Rendering speed

Page 41: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!

Challenges!• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Rendering Speed!

• MiddleVR for Unity!• Conclusion

Page 42: SAE AR/VR - The challenges of creating a VR application with Unity

Application

Page 43: SAE AR/VR - The challenges of creating a VR application with Unity

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

MiddleVR for Unity!• Conclusion

Page 44: SAE AR/VR - The challenges of creating a VR application with Unity

• Build hardware

To create and maintain presence, you, the author, need to ...

• Build software

• Build user experience

Page 45: SAE AR/VR - The challenges of creating a VR application with Unity

“VR is mainly a software problem”!Jaron Lanier

Page 46: SAE AR/VR - The challenges of creating a VR application with Unity

What is MiddleVR ?

• VR plugin !

• Functionalities ○Manage input devices

•3D trackers, keyboard, mouse, joystick !

○Manage display •Viewports, cameras, stereoscopy, cluster

Page 47: SAE AR/VR - The challenges of creating a VR application with Unity

Desktop

Page 48: SAE AR/VR - The challenges of creating a VR application with Unity
Page 49: SAE AR/VR - The challenges of creating a VR application with Unity

http://www.youtube.com/watch?v=EEKH-1EilXs

MiddleVR tutorial

Page 50: SAE AR/VR - The challenges of creating a VR application with Unity
Page 51: SAE AR/VR - The challenges of creating a VR application with Unity

Device management

Page 52: SAE AR/VR - The challenges of creating a VR application with Unity

Device management

Page 53: SAE AR/VR - The challenges of creating a VR application with Unity

Supported Devices• Via VRPN :

• A.R.T • Vicon • Optitrack • Wiimote !

• Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers,

• Ascension Flock of Bird • Intersense IS600, IS900 • 3dTech HiBall-3000 Wide Area Tracker, • Advanced Realtime Tracking Gmbh DTrack

Client, • World Viz Precision Position Tracker PPT

1.2, • Natural Point Optitrack Rigid Body Toolkit, • Phasespace optical tracking OWL, • ...

• Native drivers • Oculus Rift • Leap Motion • Kinect (Microsoft SDK) • PNI SpacePoint

Fusion • GameTrak • Razer Hydra • Motion Analysis (beta)

Page 54: SAE AR/VR - The challenges of creating a VR application with Unity

First integration of Haption’s haptic devices

http://www.youtube.com/watch?v=lnoE_VclnOI

Page 55: SAE AR/VR - The challenges of creating a VR application with Unity

Stereoscopy

• Active stereoscopy (OpenGL Quad-Buffer)!

• Which Unity can’t natively handle!

• Passive stereoscopy

Page 56: SAE AR/VR - The challenges of creating a VR application with Unity

Cluster & Multi-pipe

• Scenelock!

• Swaplock!

• Genlock!

• Multi-pipe (soon)

Page 57: SAE AR/VR - The challenges of creating a VR application with Unity

Supported displays

• Virtually any HMD • VR-Wall, Powerwall • Workbench, Holobench • HoloStage • CAVEs • 3D TVs • zSpace • Igloo Vision • …

Page 58: SAE AR/VR - The challenges of creating a VR application with Unity

Deployment

• Abstractions!

• Devices!

• 3D Nodes (User)!

• Interactions (soon)

Page 59: SAE AR/VR - The challenges of creating a VR application with Unity

Device management

Driver Tracker 1

Application

Driver Tracker 2

Page 60: SAE AR/VR - The challenges of creating a VR application with Unity

Device abstraction

Virtual Device

Application

Driver Tracker 1

Driver Tracker 2

Page 61: SAE AR/VR - The challenges of creating a VR application with Unity

Device management

Driver

Virtual Device Application

Simulation

Replay

Page 62: SAE AR/VR - The challenges of creating a VR application with Unity

Virtual Device 1

Hand (3D Node) Application

Virtual Device 2

Inverse Kinematics

User abstraction

Page 63: SAE AR/VR - The challenges of creating a VR application with Unity

Interactions• MiddleVR includes high-level interactions!

• Navigation!

• Selection!

• Manipulation!

• … More to come!

• Future: Interactions abstractions!

• Dynamically change interactions based on hardware

Page 64: SAE AR/VR - The challenges of creating a VR application with Unity

Interactions abstraction?Grab-air

Navigation

Application

Point

Steering

Selection

Manipulation

Raycasting

Go-go

Hand

Homer

Hand

WIM

Page 65: SAE AR/VR - The challenges of creating a VR application with Unity

Menus

• Ability to create menus and GUIs in HTML5 (soon)

Page 66: SAE AR/VR - The challenges of creating a VR application with Unity

Speed• Unity!

• Many options for optimizations!

• Occlusion culling!

• Static batching!

• Lightmap baking!

• etc.!

• 32-bit editor only!

• But player is 32-bit & 64-bit !!

• Subdivide parts in editor!

• Compositing!

• Cloud

Page 67: SAE AR/VR - The challenges of creating a VR application with Unity

Compositing Stereoscopy

Page 68: SAE AR/VR - The challenges of creating a VR application with Unity

Compositing Image decomposition

Page 69: SAE AR/VR - The challenges of creating a VR application with Unity

VR in the Cloud

• Temporary increase in power!

• Combine with compositing

Page 70: SAE AR/VR - The challenges of creating a VR application with Unity

Perceptive illusions

• Playing with limitations of human perception!

• Sensitivity of senses!

• Selectivity of attention!

• Useful to lower cost/complexity of VR systems

Page 71: SAE AR/VR - The challenges of creating a VR application with Unity

Perceptive illusions

(c) Franck Steinicke - Wurzburg University

Redirected walking

Page 72: SAE AR/VR - The challenges of creating a VR application with Unity

Change blindness :

(c) Evan Suma - USC/ICT

Perceptive illusionsChange blindness

Page 73: SAE AR/VR - The challenges of creating a VR application with Unity
Page 74: SAE AR/VR - The challenges of creating a VR application with Unity
Page 75: SAE AR/VR - The challenges of creating a VR application with Unity
Page 76: SAE AR/VR - The challenges of creating a VR application with Unity

Conclusion!

!

Thank you ! Questions ?

[email protected]