CS294-137: Theory and Applications of Virtual Reality and Immersive Computing

Achin Bhowmik, James O’Brien, Allen Y. Yang

Fall, 2017

Lecturers

AchinBhowmik

CTO&EVP,StarkeyHearingTechnologies

FormerVP,PerceptualComputing,

Intel

Email:achintya.k.bhowmik@gmail.com

JamesO’Brien

Professor,EECS

ChiefScientist,Avametric

Email:job@berkeley.edu

AllenY.Yang

ChiefScientist,FungInstitute

FormerCTO,AtheerLabs

Email:yang@eecs.berkeley.edu

Other Guest Lecturers: CAC.berkeley

Other Resources @ Berkeley

Other Resources @ Berkeley

l  Founded in Spring 2015 l  >200 members in each

semester enrollment

VR@Berkeley

Course ScheduleWeek1(8-23):IntroductionandCapstoneOptionsWeek2(8-30):HumanPerceptionintheContextofVRWeek3(9-6):BasicUnity3D/VRProgrammingWorkshopWeek4(9-13):CourseprojectproposalpresentationWeek5(9-20):ComputerGraphicsrelatedtopicsWeek6(9-27):3DVisionrelatedtopicsWeek7(10-4):OpticsandDisplaytechnologiesWeek8(10-11):LocalizationandMapping****************Week9:(10-18)VRFilmMaking(RichardHernandez)Week10(10-25):Gaming(JackMcCauley)Week11(11-1):Telemedicine(RuzenaBajcsy/GregorijKorillo)Week12(11-8):AR/VRinArts&Design(TedSelker)Week13(11-15):ComputationalImagingforVR(RenNg)Week14(11-22):NoclassWeek15(11-29):FinalprojectpresentationWeek16(12-6):Finalprojectpresentation

Grading PolicyYourfinalgradeswillbedeterminedbyfourfactors:1.   Attendancebyyou(10%)

2.   Interactionduringtheclassgivenbythelecturers(10%)

3.   Course/capstoneprojectpresentation(40%)

4.   Aresearchpaperdiscussingyourprojectanditsrelevantliteratureandcommercialapplications(40%)

Recommended Reading Material•  Perception:SensationandPerception

byBruceGoldstein

•  VirtualReality:VirtualRealityByStevenLaValle(andcheckouthisYouTubelectures)

•  ComputerGraphics:FundamentalsofCGbyPeterShirley

•  ComputerVision:AnInvitationto3-DVisionbyYiMa,etal.•  Display:MobileDisplays

byAchinBhowmik,etal.

•  AR/VRMarketResearch:Virtual&AugmentedReality,understandingtheraceforthenextcomputingplatformbyGoldmanSachs

Goal of the Course•  UnderstandthefundamentaltheoriesthatenableVR/ARandIC

•  UnderstandthemaintechnologydriversofVR/ARandICmarkets

•  Becomeanexpertincriticizingthecurrentsoftwareandhardwaresolutions

•  Beingabouttheinventnewsolutionsthataddressexistingneeds/painpointsofVR/ARandICapplications

•  (Optional)Beingmotivatedtopursueacareerinrelevantresearchorentrepreneurialfields

AR/VR: A Hot Market in 2016

Holodeck: A VR Experience

Leia’s Hologram: An AR Experience

Brief History of VR/AR/IC

PhotographyGraphics Stereoscopy

Interaction3DAudio

Early Forms of Paintings and Arts

CavepaintingsSince35,000–40,000BP

Writingandlanguages,Since3100BC

Early Forms of Photography

CameraObscura,circa400BC Firstphotoonpaper,1800s Kodachrome,1935

3D Illusion with Perspective

AmesRoom,byAdelbertAmes,Jr.,1946

Seeing 3D from Stereo

Wheatstonemirrorstereoscope,1838 Holmesstereoscope,1861

Audio: From Mono to Spatial 3D

Edisoncylinderphonograph,1899

DolbyStereo,1977

Inventionofheadphones,1910s

DolbyAtmosforVR

Interaction Modalities

Keyboard,since1860s Mouse,DouglasEngelbart,1963

iPhone,2007

Hands as 3D Input Device

Sensorama: The First VR Prototype

MortonHeilig,1958

Virtuality: Dawn of VR Gaming

Tilt Brush in Virtual 3D

CES 2017: Samsung 4D VR Experience

IvanSutherland,1968

The First AR Prototype

Anatomy of an AR Device: HoloLens

Definition: Virtual Reality•  VR is a computer technology that uses head mounted

displays, sometimes in combination with other sensory devices, to generate realistic images, sounds, and other sensations (touch, smell, motion, etc.) that simulate a user’s physical presence in a virtual environment.

27

Definition: Augmented Reality•  AR is a computer technology that augments a

physical, real-world environment directly or through its indirect view computer-generated sensory information, including graphics, video, and sound. AR may alter a user’s view of reality, and may also enhance one’s perception of reality.

28

Enabling Technologies/ Open Research

•  Near-EyeDisplaysandOptics

•  3DLocalization

•  3DContentCapturing

•  NewHuman-ComputerInterface

Near-Eye Optical Module

HMD Stereo Display Challenge

AccommodationVergence

Localization via Beacons

Localization via Depth Perception

Google Tango Inside-Out Localization

360 VR Capturing: Photo Stitching

GoogleJumpVR

From 360 VR to Real-Time 3D VR

Lytro Immerge

New 3D Human-Computer Interface

New 3D Human-Computer Interface

Connecting AR/VR and Robotics

Course/Capstone Projects Examples

Berkeley OpenARK

•  Lumus •  PMD•  Webcam•  Motion Sensor•  Microsoft Surface Pro•  <$5K per unit

Beta released early 2017

2 Allow for breathing room

ISAACS: Immersive Semi-Autonomous Aerial Command System

2 Allow for breathing room

Drone Fleet Control (Tomlin)

2 Allow for breathing room

Drone Safety Control (Tomlin)

VR in Autonomous Driving (Borrelli)