Wearable Displays with Focus Cues · Vision-correcting Display! printed transparency! iPod Touch...

Wearable Displays with Focus Cues!

Gordon Wetzstein!Stanford University!

EE367/CS448I: Computational Imaging and Display!stanford.edu/class/ee367!

Lecture 16!

remote control of vehicles, e.g. drones!

architecture walkthroughs!

virtual travel!education!

robotic surgery!gaming!

simulation & training! visualization & entertainment!

a trip down the rabbit hole!

VR at Stanford’s Medical School!

!•! Lucile Packard Children’s Hospital: used to

alleviate pain, anxiety for pediatric patients!!!•! VR Technology Clinic: applications in

psychotherapy, mental health, for people with phantom pain, …!

•! help train residents, assist surgeons planning operations, …!

photo from Stanford Medicine News!

Exciting Engineering Aspects of VR/AR!

•! sensors & imaging!•! computer vision!•! scene understanding!

•! photonics / waveguides!•! human perception !•! displays: visual, auditory,

vestibular, haptic, …!•! VR cameras!

•! cloud computing!•! shared experiences!

•! HCI!•! applications!

•! compression, streaming!

•! CPU, GPU!•! IPU, DPU?!

Exciting Engineering Aspects of VR/AR!

•! CPU, GPU!•! IPU, DPU?!

•! sensors & imaging!•! computer vision!•! scene understanding!

•! VR cameras!

•! cloud computing!•! shared experiences!

•! compression, streaming!

•! photonics / waveguides!•! human perception !•! displays: visual, auditory,

vestibular, haptic, …!

•! HCI!•! applications!

Where We Want It To Be!

image by ray ban!

Personal Computer!e.g. Commodore PET 1983!

Laptop!e.g. Apple MacBook!

Smartphone!e.g. Google Pixel!

AR/VR!e.g. Microsoft Hololens!

A Brief History of Virtual Reality!

1838! 1968! 2012-2017!

Stereoscopes!Wheatstone, Brewster, …!

VR & AR !Ivan Sutherland!

VR explosion!Oculus, Sony, HTC, MS, …!

Nintendo!Virtual Boy!

Ivan Sutherland’s HMD!•! optical see-through AR, including:!

•! displays (2x 1” CRTs)!

•! rendering!•! head tracking!

•! interaction!

•! model generation!

•! computer graphics!

•! human-computer interaction!

I. Sutherland “A head-mounted three-dimensional display”, Fall Joint Computer Conference 1968!

Nintendo Virtual Boy!

•! computer graphics & GPUs were not ready yet!!!

Game: Red Alarm!!

Where we are now!

IFIXIT teardown!

Virtual Image!

1d+ 1d '

d’!f!

Problems: !!•! fixed focal plane!

•! no focus cues !!

•! cannot drive accommodation with rendering!!

Focus Cues – An Important Depth Cue!

Duane, 1912!

Age (years)!8! 16! 24! 32! 40! 48! 56! 64! 72!

4D (25cm)!

8D (12.5cm)!

12D (8cm)!

Importance of Focus Cues Decreases with Age - Presbyopia!

0D ("cm)!

16D (6cm)!

Relative Importance of Depth Cues!C

Blur Gradient Driven Accommodation!

The Vergence-Accommodation Conflict (VAC)!

Real World:!Vergence &!

Accommodation! Match!!

Current VR Displays:!Vergence & !

Accommodation!Mismatch!

•  Visual discomfort (eye tiredness & eyestrain) after ~20 minutes of stereoscopic depth judgments (Hoffman et al. 2008; Shibata et al. 2011)!

!•  Degrades visual performance in terms of reaction times and acuity

for stereoscopic vision (Hoffman et al. 2008; Konrad et al. 2016; Johnson et al. 2016)!

•  also: double vision (diplopia), reduced visual clarity, possibly nausea!

Consequences of Vergence-Accommodation Conflict!

•  Q1: How to address the vergence-accommodation conflict for users of different ages?!

•  Q2: Can computational displays effectively replace glasses in VR/AR?!

•  Q3: What are (in)effective near-eye display technologies?!

possible solutions: gaze-contingent focus, light fields, …!

1. Gaze-contingent Focus!

Magnified Display!Display!

Fixed Focus!

1d+ 1d '

d! d’!

Adaptive Focus!

1d+ 1d '

actuator " vary d’!

Adaptive Focus!

focus-tunable!lens " vary f!

1d+ 1d '

Adaptive Focus - History!

•! M. Heilig “Sensorama”, 1962 (US Patent #3,050,870)!•! P. Mills, H. Fuchs, S. Pizer “High-Speed Interaction On A Vibrating-Mirror 3D Display”, SPIE 0507 1984!•! S. Shiwa, K. Omura, F. Kishino “Proposal for a 3-D display with accommodative compensation: 3DDAC”, JSID 1996!•! S. McQuaide, E. Seibel, J. Kelly, B. Schowengerdt, T. Furness “A retinal scanning display system that produces multiple focal planes

with a deformable membrane mirror”, Displays 2003!•! S. Liu, D. Cheng, H. Hua “An optical see-through head mounted display with addressable focal planes”, Proc. ISMAR 2008!

manual focus adjustment!Heilig 1962!

automatic focus adjustment!Mills 1984!

deformabe mirrors & lenses!McQuaide 2003, Liu 2008!

Padmanaban et al., PNAS 2017!

sstereoscopic

distancevirtual imagedistance

virtual imagedistance

stereoscopic distance

stereoscopicdistance

vergence!accommodation!

Conventional Stereo / VR Display!

vergence!accommodation!

Removing VAC with Adaptive Focus!

Follow the target with your eyes!

4D !(0.25m)!

0.5D!(2m)!

Stimulus!

Accommodative Response!

Time [s]!

Stimulus!Accommodation!

n = 59, mean gain = 0.29!

Accommodative Response!Co

Time [s]!

Stimulus!

Accommodative Response!

Time [s]!

Stimulus!Accommodation!

Accommodative Response!Co

sstereoscopic

distancevirtual imagedistance

Time [s]!

n = 24, mean gain = 0.77!

Do Presbyopes Benefit from Dynamic Focus?!G

conventional!

conventional!dynamic!

Response for Physical Stimulus!Heron & Charman 2004!

Gaze-contingent Focus!•! non-presbyopes: adaptive focus is like real world, but needs eye tracking!!

lens!micro

display!

virtual image!

eye tracking!

Gaze-contingent Focus!

at ACM SIGGRAPH 2016!at ACM SIGGRAPH 2016at ACM SIGGRAPH 2016at ACM SIGGRAPH 2016

Oculus announces gaze-contingent varifocal display at F8, 05/2018 !

Oculus Half Dome Prototype!

Video courtesy of Facebook/Oculus!

Summary!

•  adaptive focus drives accommodation and can correct for refractive errors (myopia, hyperopia)!

•  gaze-contingent focus gives natural focus cues for non-presbyopes, but require eyes tracking!

•  presbyopes require fixed focal plane with correction!

2. Light Field Displays!

Near-eye Light Field Displays!

Idea: project multiple different perspectives into different parts of the pupil!!

Light Field Cameras!Light Field CamerasLight Field Stereoscope!

Huang et al., SIGGRAPH 2015!

Backlight!

Thin Spacer & 2nd panel (6mm)!

Magnifying Lenses!LCD Panel!

Light Field Stereoscope!

Target Light Field!Input: 4D light field for each eye!

Multiplicative Two-layer Modulation! Input: 4D light field for each eye!

Multiplicative Two-layer Modulation!Multiplicative Two-layer Modulation

Reconstruction:!Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:Reconstruction:for layer t1!

Tensor Displays, !Wetzstein et al. 2012!

Input: 4D light field for each eye!

Traditional HMDs!- No Focus Cues!

The Light Field HMD!Stereoscope!

Tensor Displays!

Wetzstein et al., SIGGRAPH 2012!

Vision-correcting Display!

iPod Touch prototype !printed transparency!Huang et al., SIGGRAPH 2014!

prototype!

300 dpi or higher!

Diffraction in Multilayer Light Field Displays!Wetzstein et al., SIGGRAPH 2011!Lanman et al., SIGGRAPH Asia 2011!Wetzstein et al., SIGGRAPH 2012!Maimone et all., Trans. Graph. 2013!…!

Hirsch et al, SIGGRAPH 2014!

Less diffraction artifacts with LCoS !

blur!!

3. Accommodation-invariant Near-eye Displays!

Blur Gradient Driven Accommodation!

PSF Engineering!

Q: can we drive accommodation with stereoscopic cues by optically removing the retinal blur cue?!

How do we remove the blur cue?!

Aperture Controls Depth of Field!

Image courtesy of Concept One Studios!

Maxwellian-type (pinhole) Near-eye Displays!

Point Light!Source!

Maxwellian-type (pinhole) Near-eye Displays!

Severely reduces eyebox; requires dynamic steering of exit pupil

Spatial Light Modulator!

Point Light!Source!

Focal Sweep!

EDOF Cameras:!Dowski & Cathey, App. Opt. 1995!Nagahara et al., ECCV 2008!Cossairt et al., SIGGRAPH 2010!

Convolution!

Deconvolution!

Target Image!Target!

Target Image! Conventional Display @ 1D!Target!

Conventional!

Target Image! Conventional Display @ 3D!Target!

Conventional!

Target!

Conventional!

AI @ 3D! Conventional Display @ 3D!

Stimulus!

Accommodation Invariant

0 5 10 15 20Time (s)

Gain: 0.61

Dynamic

0 5 10 15 20Time (s)

Gain: 0.850 5 10 15 20

Time (s)

Measured User Response!

0 5 10 15 20Time (s)

Gain: 0.61

Dynamic

0 5 10 15 20Time (s)

Gain: 0.85

Conventional

0 5 10 15 20Time (s)

Gain: 0.35

StimulusAverage

Measured User Response!

0 5 10 15 20Time (s)

Gain: 0.61

Dynamic

0 5 10 15 20Time (s)

Gain: 0.85

Conventional

0 5 10 15 20Time (s)

Gain: 0.35

StimulusAverage

Future: multifocal lenses!Now: benchtop!

Photonics Challenges for Getting Here!

Thin Beam Combiner?!

Thin Beam Combiner!!

Pepper’s Ghost 1862!

Case Studies!

Google Glass!

Meta 2!

•! larger field of view (90 deg) than Glass!

•! also larger device form factor !

Microsoft HoloLens!

•! diffraction grating-based waveguide!

•! LCoS microdisplay!!•! field of view: 34°

diagonally, 16:9 aspect, 47 pixels per visual degree!

Microsoft HoloLens 2!

Wall et al. US 10,025,093 2018

•! laser-scanned waveguide display!

•! claimed 2K resolution per eye (2560x1440), probably via “interlaced” scanning!

•! field of view: 52° diagonally (3:2 aspect, 47 pixels per visual degree) !

Zeiss Smart Optics!•! great device form factor!•! polycarbonate light guide – easy to manufacture and robust!•! smaller field of view (17 deg)!

Challenges: Eye Box vs Field of View!

eye box / exit pupil! entrance pupil!entrance pupil

•! need small entrance pupil (small device) and large exit pupil (large eye box) - pupil needs to be magnified!

•! need small display (small device) but large field of view – image needs to be magnified!

field of view!

•! pupil needs to be magnified!•! image needs to be magnified!

field of view!

can’t get both at the same time – etendue!!

•! possible solutions: exit pupil replication (loss of light), live with small FOV (not great), dynamically steer eye box (mechanically difficult), ..!

field of view!

Challenges: Chromatic Aberrations!

•! thin grating couplers create chromatic aberrations!

Challenges: Chromatic Aberrations!

•! all solutions have their own problems: ease of manufacturing, yield, robustness, cost, …!

volume holographic couplers, e.g. TruLife Optics!

stacked waveguides!

Occlusions!Case 1: !

virtual in front of real!Case 2: !

real in front of virtual!

virtual!object!

real!object!

" difficult: need to block real light!!

" easy: don’t render virtual object everywhere!

Next quarter: EE 267!

The End!

Wearable Displays with Focus Cues · Vision-correcting Display! printed transparency! iPod Touch...

Documents

JOURNAL OF HEMATOLOGY...JOURNAL OF HEMATOLOGY & ONCOLOGY Induction of acquired drug resistance in endothelial cells and its involvement in anticancer therapy Huang et al. Huang et

Effect of NO on 1,3,5-trimethylbenzene (TMB ... - ACP · tions (Paulsen et al., 2005; Rodigast et al., 2017; Wyche et al., 2009; Huang et al., 2015; Sato et al., 2012). They re-ported

(io) Patent No.: US 9,586,192 B2 Huang et al

Partially melted, mica-bearing crust in Central Tibetciei.colorado.edu/.../Hackeretal_Tectonics_2014.pdfanisotropy [e.g., Shapiro et al., 2004;Duretetal., 2010;Huang et al., 2010;Xie

Huang HY et al. The efficacy and safety of multivitamin and mineral

PROTOTYPE - ddbunlimited.comprototype prototype prototype prototype prototype prototype prototype prototype prototype prototype prototype prototype f e g h f e g h scale: 1:12 4. 4.875

Étude et réalisation d’un prototype d’archivage

pure.southwales.ac.uk · Web viewmajority of which were described by Huang et al (Huang et al., 2007). Bipolar membrane electrodialysis requires expensive, bipolar, water-splitting,

Huang et al:MTG-IRS OSSE. EMC seminar, 1/11/2008. 1 MTG-IRS: An Observing System Simulation Experiment (OSSE) on regional scales Xiang-Yu Huang, Hongli

Inconsistent Ontology Diagnosis: Framework and Prototype - VUhuang/sekt/sekt361.pdfStefan Schlobach and Zhisheng Huang (Vrije Universiteit Amsterdam) Abstract. EU-IST Integrated Project

Huang Jian, Lin Tianxin, Xu Kewei, Huang Hai, Yao Yousheng et al Urology Department

Prototype CERN-option: simulation results Maria Elena Martin Albarran UCL et al

Automation for Food Engineering - Y. Huang, et al., (CRC, 2001) WW

Coleus aromaticus leaf extract mediated synthesis of ... · 2010; Huang et al. 2007; Daizy 2011), seeds (Bar et al. 2009b),fruits(Dubey et al. 2010;Jain etal. 2009),latex (Bar et

Deep Learning. Why? Source: Huang et al., Communications ACM 01/2014

Prototype Prototype

Huang et al Reliability of measurement of skin ultrasonicira.lib.polyu.edu.hk/bitstream/10397/2322/1/Huang et al Reliability of... · 5 changes of acoustic scatterers in biological

Jikun Huang, Ruifa Hu Center for Chinese …...constraints to further development of plant biotechnology in China. The work of Keeley (2003), Huang and Wang (2003) and Huang et al

Python/C FASE prototype L. Paioro, B. Garilli et al. OPTICON Network 9.2 MiMa Collaboration PANDORA @ INAF-IASF Milano L. Paioro - Python/C FASE prototype

`` Je suis le prototype '' : femmes bioniques et