Madeline WrableGIS Specialist

CLSTL’19

pngtree.com 1

Content

• GIS & Data Lab

• VR Pilot Project Story

• What is VR

• VR hard/software

• Getting started

pngtree.com 2

Content

• GIS & Data Lab

• VR Pilot Project Story

• What is VR

• VR hard/software

• Getting started

pngtree.com 3

Getty Images 4

Massachusetts Institute of Technology

Getty Images 5

Rotch Library

• GIS Lab• Additional software• GIS data repository, GeoWeb• One-on-one help• Workshops

MIT GIS Services: GIS & Data Lab

Credit: Lisa Sweeney 6

Content

• GIS & Data Lab

• VR Pilot Project Story

• What is VR

• VR hard/software

• Getting started

pngtree.com 7

Adapted from “Reality Bytes: VR and AR” –Matt Bernhardt, MIT Libraries

8

VR Pilot Project Story

Jesussanz/Getty Creative 8

Adapted from “Reality Bytes: VR and AR” –Matt Bernhardt, MIT Libraries

9Jesussanz/Getty Creative 9

The MIT Libraries Experimental Collections Fund Supports projects using library

datasets in new ways.

I used it for a drone-pilot-project. Do you have any ideas?

VR Pilot Project Story

Adapted from “Reality Bytes: VR and AR” –Matt Bernhardt, MIT Libraries

10

The MIT Libraries Experimental Collections Fund Supports projects using library

datasets in new ways.

I used it for a drone-pilot-project. Do you have any ideas?

Jesussanz/Getty Creative 10

I wonder if virtual reality (VR) would be a good fit…

VR Pilot Project Story

Content

• GIS & Data Lab

• VR Pilot Project Story

• What is VR

• VR hard/software

• Getting started

pngtree.com 11

Facebook F8 developer’s conference, 2015 12

History of Immersive Reality

Breakthroughs

21

Results and Future Directions

34

oculus quest

image: facebook

Content

• GIS & Data Lab

• VR Pilot Project Story

• What is VR

• VR hard/software

• Getting started

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

• Google Cardboard

• Google Daydream

• Samsung Gear VR

46

• Oculus Rift

• HTC Vive

• Sony PlayStation VR

• FOVE VR

• Google Hololens

• …

Content Viewing Platform Examples

Mobile Devices

Laptop/Desktops

Game Consoles

Standalone HMDs & Eye Tracking

…

Mobile Device: Google Cardboard

• ~$15 simple headset

• Free or paid smartphone apps

• “Head in a beachball” content

• Phone gyroscopes used to determine view direction

• Only click in center of your vision

• https://vr.google.com/cardboard/developers/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Mobile Device: Google Daydream

• ~$80 less simple headset

• Free or paid smartphone apps

• “Head and hand in beachball”

• Hand-held wand to separate looking and clicking

• https://developers.google.com/vr/daydream/overview

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Mobile Device: Samsung Gear VR

• ~$100, requires a Samsung phone

• Free or paid smartphone apps

• Comes with handheld remote

• Limited to Oculus software library

• http://developer.samsung.com/stream-code101/gear-vr

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Personal Computer: Oculus Rift

• ~$500 hardware kit

• Powerful computer required

• Free or paid software applications

• Cubicle-sized space

• One body sensorsHandheld wand or controllerHeadset with speakers

• https://developer.oculus.com/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Personal Computer : HTC Vive

• ~$800 hardware kit

• Powerful computer required

• Free or paid software applications

• Cubicle or Room-sized space

– Two body sensors

– Two handheld wands

– Headset with speakers

• Note: now has a new version (Pro)

• https://developer.viveport.com/us/develop_portal/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Console: Sony PlayStation VR

• $300 headset & $300 console

• 45 million PlayStation 4 owners makes this an accessible path to VR

• Easy adoption

• https://www.playstation.com/en-us/develop/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Standalone: FOVE VR

• ~$600 hardware kit

• Founded by a former game producer at Sony

• Headset with eye tracking

• https://www.getfove.com/developers/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Standalone: HoloLens

• ~$3000 hardware kit

• Mixed Reality

• Development kits started selling February 29, 2016

• https://developer.microsoft.com/en-us/windows/mixed-reality/install_the_tools

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

full reality

full virtuality

full reality

full virtuality

mixed reality

full reality

full virtuality

low presence

highpresence

full reality

full virtuality

tethered

untethered

low presence

highpresence

full reality

full virtuality

low presence

highpresence

vive

windows vr

gear vr

cardboard

oculus dk1

smartphonesmartphone

oculus dk2

oculus rift

daydream

google glass

hololens

meta 2

magic leap

smartphone

pimax vr

oculus quest

htc vive cosmos

vive pro eye

tethered

untethered

full reality

full virtuality

low presence

highpresence

vive

windows vr

gear vr

cardboard

oculus dk1

smartphonesmartphone

oculus dk2

oculus rift

daydream

google glass

hololens

meta 2

magic leap

smartphone

pimax vr

oculus quest

htc vive cosmos

vive pro eye

tethered

untethered

tracking optics / displays rendering telecomother

senses

no tracking: 3DOFV

external tracking: 6DOFV

inside out tracking: 6DOFV

inside out tracking + location tracking: 6DOF + coordinates

larger FOV (~ 200°)greater resolution

greater refresh rateopto electronics

variable focus lenseseye tracking

greater processormore efficient GPU

ray tracingfoveated rendering

optimization of rendering pipeline

new 3D graphic techniquesAI in rendering processes

AI in content creation

greater bandwidthV

ultra low latency wireless local

Vultra low latency

wireless universal

better ergonomic haptic interfacesdynamic binaural

audioother senses inputsgeneral locomotion

+ better batteries

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

Content Creation Platforms

• Photo Spheres

• Photogrammetry

• 3D Scanning

• 3D Modeling

• Game Engines

• …

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Photo Sphere: Ricoh Theta

• ~$300 digital camera

• Twin fisheye lenses

• Still photographs or video

• Includes streaming capability

• Suitable for Cardboard or Daydream-style viewers

• Output supported by Facebook, Flickr, YouTube, etc

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Photo Sphere: VUZE

• ~$800 digital camera

• Four pairs of fisheye lenses

• Still photographs or video

• Stereo 3d photo spheres

• Suitable for Cardboard or Daydream-style viewers

• Output suitable for YouTube

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Photogrammetry: Agisoft PhotoScan

• $179 Standard & $3,500 Pro (license for only one computer)

• Able to reconstruct 3D models from drone acquired photos

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

Photogrammetry: ArcGIS Drone2Map

• Prohibitively expensive without educational licensing, even then ~$1,000 per seat

• Part of a spatial software suite

• Somewhat lacking in 3D model capabilites

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

3D Scanning: Sense

• $400 3d scanner

• Time of flight & white light scanner

• Useful for generating 3d models to import into VR/AR content

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

capturing modeling renderingagisoft photoscan360 video suiteAdobe Premiere...

Autodesk RevitSolidworksSketchUpRhinocerosAutodesk 3DS MaxAutodesk MayaStrataBlenderCinema 4D...

Unreal EngineCryEngineUnity...

3D Modeling Software

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

capturingrepresenting

interactive

non interactive

synthetic simulations

immersive video

photogrammetryvideogrammetry3D scanning

CGI / Digital Animation

hand built AI builtproceduraly built

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

Game Engine: Unity

• Free for hobbyists and beginners

• All purpose game engine

• Poor internal editing options

• https://unity3d.com/learn/tutorials/topics/virtual-reality/getting-started-vr-development

Game Engine: Unreal Engine

• Free up to $3,000 revenue, then charges a 5% royalty

• Game engine authored by Epic Games (BioShock, Assassin’s Creed)

• https://docs.unrealengine.com/latest/INT/Platforms/index.html

Game Engine: Cry Engine

• $10/month but no royalty fees

• Game Engine designed by the German developer Crytek (Far Cry)

• Licensed by Amazon

• http://docs.cryengine.com/pages/viewpage.action?pageId=25536773

SoftwareInterface

3D/VR

ContentCreation

2D 3D

Content Viewerheadsets

+

VR Pipeline

Content Creation Software

Content Creation

2D 3D

VR Pipeline

+

Content Viewer

VR Pipeline Future Direction

Content Creation Platforms

Content Viewing Platforms

Software Development Kits (SDKs)

Content

• GIS & Data Lab

• VR Pilot Project Story

• What is VR

• VR hard/software

• Getting started

• GIS Lab• Additional software• GIS data repository, GeoWeb• One-on-one help• Workshops

MIT GIS Services

• GIS Lab » powerful computers & graphics cards (content viewer)• Additional software » 3D modeling software (content creation)• GIS data repository, GeoWeb » available datasets (content)• One-on-one help » exposure to the community• Workshops » teaching experience

• Perfect for a VR-pilot-project

MIT GIS Services

• GIS Lab » powerful computers & graphics cards (content viewer)• Additional software » 3D modeling software (content creation)• GIS data repository, GeoWeb » available datasets (content)• One-on-one help » exposure to the community• Workshops » teaching experience

• Perfect for a VR-pilot-project

MIT GIS Services+ headset

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

VR Applications

• Expensive: If it would cost a lot to do something in real life, like visit the statue of David in Florence, Italy, it might make sense to do it virtually.

• Dangerous: Climbing Mount Kilimanjaro might be the adventure of a lifetime, but it could also be your last. Several people die on its slopes every year.

• Impossible: You can’t travel back in time, grow a third arm, or experience life as a person of a different race or gender. But VR can give you a surprisingly visceral taste of what it would be like if you could.

• Rare: You could go whale-watching a dozen times without seeing a humpback breach right next to your boat. Or you could do it once in VR.

Jeremy Bailenson:On good candidates for VR

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries

image: mike kelley

image: ubisoft

image: gary wright

MIT GIS Lab: 3D & VR Applications

Pixabay.com

Applications Overview

Drone

Floor plan

VR Software&Hardware3D SoftwareSource Imagery

Kisspng.com, MIT Libraries, Dreamstime.com, & Pixabay.com

The GIS Lab’s drone collected imagery of geological formations near Death Valley National Park, CA.

Pixabay.com

Credit: Daniel Sheehan

Point Cloud Solid Mesh Textured Mesh

Processed imagery using Agisoft Photoscan software:

Credit: Daniel Sheehan

Textured mesh imported into Unity

Clicking the center button allows teleporting &Pulling the trigger button allows flying

Useful for those who:• couldn’t go to the field • would like to review findings• analyze the 3D model further

Credit: Madeline Wrable

HTV Vive Controller

Applications Overview

Drone

Floor plan

VR Software&Hardware3D SoftwareSource Imagery

Kisspng.com, MIT Libraries, Dreamstime.com, & Pixabay.com

Started with a floor plan in AutoCAD

Credit: Madeline Wrable

Traced 3D shapes from floorplan…

…and extruded them in AutoCAD

Credit: Madeline Wrable

Exported 3D model as an .fbx file and imported it into Unity

Clicking center button allows teleportingUseful for space planning and lab design

Credit: Madeline Wrable

Workflow Resources

Drone• Drone model: Phantom 4

• Flight planning: DroneDeploy

• Photo stitching: PhotoScan

Floorplan

• Exported as FBX files, and imported into Unity

• Used SteamVR & Virtual Reality Toolkit (VRTK)

• Floorplans : MIT Facilities

• 3D software: AutoCAD

Credit: Madeline Wrable

Content Creation Software

for display in headset

& follow the tutorials available for VR

Download software, most are free for educational purposes

Content Creation

2D 3D

VR Pipeline

+

Content Viewer

Unity

Unreal

CryEngine

VirtualRealityToolkit (VRTK)

Rhino

Blender

SketchUp

Autodesk

Future applications

• Launch XR checkout • AR/MR applications• Create 360° library tour• Invite community experts• More learning materials

Pixabay.com

Tips & Tricks

• Find where you already have some equipment (e.g. computer, graphics card, software)

• Make sure whatever plug comes off the headset plugs directly into your device (phone/computer)

• Explore other’s apps. & watch VRTK tutorials

• Think of a simple application to get started with

• Set a date to teach others & make learning materials

32

Game Design Examples

115

Unity Demo

121