8D: Interacting with a Relightable Glasses-Free 3D DisplayMatthew Hirsch1 Shahram Izadi2

1MIT Media Lab75 Amherst St.

Cambridge, MA, USAmhirsch,holtzman,raskar@media.mit.edu

Henry Holtzman1 Ramesh Raskar1

2Microsoft Research UK7 JJ Thompson Avenue

Cambridge, UKshahrami@microsoft.com

Figure 1. The 8D prototype allows for glasses-free display of 3D content, whilst simultaneously capturing any incident light for re-illumination andinteraction. From left to right: A user shines a light source (lamp) at the 3D display, and the rendered 3D model is correctly re-illuminated. Virtualshadows can be cast by placing a finger between the display and light source. This allows any light source to act as an input controller, for example toallow intuitive interaction with a medical data set. 8D works by simultaneously capturing and displaying a 4D light field (as shown bottom left inset).

ABSTRACTWe present an 8-dimensional (8D) display that allowsglasses-free viewing of 3D imagery, whist capturing and re-acting to incident environmental and user controlled lightsources. We demonstrate two interactive possibilities en-abled by our lens-array-based hardware prototype, and real-time GPU-accelerated software pipeline. Additionally, wedescribe a path to deploying such displays in the future, usingcurrent Sensor-in-Pixel (SIP) LCD panels, which physicallycollocate sensing and display elements.

Author Keywords3D display; Light Fields; Light-based interaction, relightable

ACM Classification KeywordsH.5.2 Information Interfaces And Presentation: UserInterfaces: Input Devices and Strategies

General TermsAlgorithms, Experimentation, Design

INTRODUCTIONImagine a display that behaves like a window. Glancingthrough it, viewers perceive a virtual 3D scene with correctparallax, without the need to wear glasses or track the user.Light that passes through the display correctly illuminatesboth virtual objects on the display and physical objects in the

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHI’13, April 27–May 2, 2013, Paris, France.Copyright 2013 ACM 978-1-4503-1899-0/13/04...$15.00.

environment. While researchers have considered such dis-plays, or prototyped subsets of these capabilities, we con-tribute an interactive 8-dimensional (8D) display which si-multaneously captures and displays a 4D light field.

We describe the design of our lens-array based, projector-camera 8D display prototype, and GPU-based pipeline forreal-time rendering and capture of 4D light fields. Our proto-type provides horizontal and vertical parallax as a user moveswithin the viewing region, without the need for user instru-mentation or tracking. Additionally, our display simultane-ously captures the incident 4D light field from environmentalor user controlled light sources. We demonstrate the use ofsuch a display in interactive scenarios: allowing for realisticrelighting of virtual 3D objects, as well as the unique abilityto use any light source as a user input controller, for exam-ple to visualize medical imaging data. With the advent ofsensor-in-pixel (SIP) LCD displays, we propose a clear pathto implementing thin, portable, 8D displays in the future.

BACKGROUNDLight fields have long been a valuable tool in rendering [15],and more recently in developing next-generation cameras [18,21] and displays [22]. They have also seen limited use forHCI, where depth information and gesture can be extractedfrom a light field captured through a prototype SIP screenand combined mask [8].

Glasses-free 3D parallax barrier [11] and lens-array [16] dis-plays have existed for over 100 years. Nayar et al. [17] cre-ate a lighting sensitive display, though it cannot accuratelymap shadows and specularities. BRDF displays can simu-late flat surfaces with a particular Bi-Directional ReflectanceDistribution Function [9]. 6D displays that demonstrate 4Drelighting of 2D images have been shown in both active [8]

Figure 2. The GPU pipeline and associated data. The pipeline is conceptually divided into three stages: deinterlacing, light field rendering, andinterlacing. A 4D input light field is combined with a 4D output light field, synthesized from 3D model geometry. From left to right, images capturedthrough the lens array are deinterlaced using lens center positions determined by calibration. Each view of the captured light field is projected onto 3Dmodel data. Output light field views are generated by a virtual camera array, pre-filtered, and interlaced for display on the lens array.

and passive [7] modes. A recently shown 7D display [20]tracks a single light point as input. In a closely related work,Cossairt et al. [4] implement a 7fps 8D display, but focus onrendering illumination effects for a 2D camera, rather than3D perception for a live viewer. Our work contributes a hard-ware approach to real-time 8D display that is compatible withemerging display technologies and a new GPU rendering andcapture pipeline to make simultaneous, interactive 4D light-ing and 4D capture feasible.

Our display offers interesting possibilities for interactingthrough light. Light pens and widgets have been previouslyused for interaction [2]. In recent years, lighting widgetshave been integrated into tabletop computing systems [14],and novel optics and computer vision have been used for in-teraction with screens, tables, and physical surfaces over ascreen [12, 19, 6]. We are proposing the first interaction sys-tem capable of fully capturing and displaying arbitrary lighttransport within a volume above the display.

IMPLEMENTATION

HardwareWe propose to implement an 8D display by placing an ar-ray of microlenses on a SIP LCD screen. Due to the pixelpitch limits of existing SIP hardware, we implement anequivalent projector-camera system (Figure 1). We place a150mm×150mm hexagonal lens array (Fresnel Tech. #360,0.5mm pitch) atop a Grafix acetate diffuser onto which weboth image and project. This optically simulates the ortho-graphic light field produced by the SIP display.

Our prototype uses a grayscale Point Grey Gazelle 2048 ×2048, 120fps camera with a 50mm Schneider Xenoplan lensas a sensor. A Sanyo PLV-Z800 1920 × 1080 projector ismodified by shifting lens forward 4mm, allowing it to createa focused 325dpi image, matching the width of the hexag-onal lens sheet. The projector and camera share an opticalpath through a 40/60 beamsplitter. We prevent cross-talk bymultiplexing both through crossed linear polarizers.

CalibrationOur prototype system necessitates a calibration step to alignthe sample grids of the camera and projector with the real-world coordinates of the lens sheet. To calibrate the camera, a

collimated light source is placed above the lens sheet, creatinga point below each lens center, which can be located in thecamera’s view. The projector is calibrated using the moriemagnifier [10] effect. A hexagonal grid of red bars on a blackbackground is projected at the expected lens center locations.The scale and rotation of the projected image are adjusteduntil the central view above the lens sheet is solid red. Seethe supplementary video for details on both techniques.

Real-time GPU PipelineThe 8-Dimensional nature of our relightable 3D display is ap-parent within our GPU pipline implementation. Our main 8Drendering pipeline depicted in Figure 2 and described in thissection, generates both the input and output light fields usingoff-screen rendering to texture arrays. Implemented in HLSLand DirectX 11, the GPU pipeline runs in real-time on anNvidia GTX 470 GPU. Our draw loop consists of N×M ren-derings of our 3D scene, one pass for each of the N×M lightfield views shown on our display (Figure 2, Center). Eachview is observed using an off-axis oblique camera projection[13], corresponding to the view angle through the lens sheetof our 8D display prototype, and then rendered into a slice ofa 2D texture array. We implement a simplified version of therendering equation, neglecting BRDFs.

Lo(x, ω) =

∫Ω

Li(x, ω′)(−ω′ · n)dω′ (1)

where Li is the measured incident light field, Lo the displayedlight field, and ω′ the incoming lighting direction. Though inour model local regions are invariant in outgoing light direc-tion, ω, each light field view is generated with a view matrixcorresponding to a virtual skewed orthographic camera view-ing the scene from ω.

To capture a 4D light field we deinterlace images recordedfrom the back of the lens array in our GPU pipeline (Fig-ure 2, Left). For each render pass, we project P ×Q capturedinput views onto the scene using projective texture mapping.In practice, we use 5×5 views for both input and output, as alimited number of texture look-up coordinates can be passedbetween the shader stages. After the 8D rendering is com-pleted, two additional render passes with associated shaderstages implement two 4D spatio-angular filtering operations,and hexagonal interlacing/deinterlacing (Figure 2, Right).

PERCEPTION AND INTERACTION

Figure 3. A hand-held light (Left) reveals either a patient’s skin andfacial features, or brain tissue (Center). This non-physical interpretationof lighting intensity leads to a novel interface for viewing medical datasets. The captured incident light field (Right) is projected onto a headand brain model, modulating the skin transparency by incident lightintensity. (Center) One view of the resulting output light field. (Bottom)The x-ray demo under various lighting conditions.

Our goal in creating an 8D Display is to take a step towardsdisplays that can produce the convincing illusion of physicalreality [1, 5]. A key aspect of this goal will be the ability ofthese displays to realistically react to incident environmentallighting. Figure 1 shows the 8D display prototype renderinga virtual 3D model. When viewing the display, the modelappears to be 3D, with full parallax. In this example, theuser also moves a lamp over the display, and the 3D modelresponds to the incident light and is correctly re-illuminated.

In a second demonstration, depicted in Figure 3, more intenselight acts like a virtual x-ray, revealing inner structure in MRIdata. In this case, the skin of the patient is visible under lower

light conditions, allowing a clinician to visually identify thepatient. As the x-ray lamp is brought closer to the screen,the skin layer becomes transparent, revealing the segmentedbrain imagery.

LIMITATIONSOur 8D display prototype is subject to optical and computa-tional limitations on its performance. Our prototype supports7 × 7 views optically. However, due to limitations of ourGPU pipeline, we are able to support only 5× 5 views in realtime. Though our output images are resampled onto a hexag-onal grid in our GPU pipeline in order to accommodate thehexagonal lens array, the approximate equivalent rectilinearresolution of our display is 274× 154 per view. With a 3mmfocal length, the lens array offers a 19 field-of-view.

Sampling theory for automultiscopic displays, which predictsthe inherent spatio-angular resolution trade-off shared by allsuch designs, is characterized by Zwicker et al. [23]. Theirfrequency domain analysis explains the depth of field exhib-ited by automultiscopic displays. Following Zwicker, for adisplay with angular sampling rate ∆v, and spatial samplingrate ∆t, objects at distance |z| > ∆t

∆v will be blurred. Defin-ing the t plane at the focal point of the lens sheet, and the vplane at unit distance, ∆t = 0.547mm and ∆v = 0.026mm.Given the above equation for z, objects up to 21mm from thedisplay are reproduced without blur. Empirical depth of fieldcharacterization shows satisfactory reproduction for objectsextending up to 3cm. This can be improved with increasedangular resolution.

As is apparent in the video, direct reflection and scatteringfrom the lens sheet competes with the light emitted from thescreen of 8D display. Anti-reflection coatings can reducesuch reflections to less than 1% of incident light, and are com-mon in commercial optics. (E.g., Anti-Reflective Coatingsfor Acrylic by American Polarizers, Inc.)

These factors limit the spatial resolution of our prototype dis-play currently. For example, in Figure 3 (Center, vs Bottom),details of the face and brain are obscured in our lower spatialresolution display prototype. Even at these low resolutions,our prototype illustrates the potential of such light-based in-teractive displays.

FUTURE WORKThe examples demonstrated in this work only begin to touchon the possibilities enabled when light transport is modeledby a display in this way. Abundant interactive possibilitiesinclude: using an off-the-shelf light source as 6DoF inputcontroller, direct manipulation of physical lights to cause re-lighting of 3D scenes, augmented-reality mirrors, accuratelymimicking surfaces with exotic BRDFs, and applying non-realistic physics to real lighting sources. Beyond our demon-strated x-ray example, one could imagine implementing non-euclidean optics, with multiple 8D displays. 8D Displays ofsufficient intensity can be used to computationally illuminateobjects in the environment, for interaction, user guidance, oradvanced shape scanning and acquisition. Modulating ren-dering properties based on incident light color would also be a

powerful interaction metaphor, but our prototype is currentlycapable only of grayscale input.

Much of the potential impact of this research is predicatedon the existence of Sensor-In-Pixel (SIP) LCDs. In recentyears LCD manufacturers have introduced semiconductortechnologies that combine light sensitive elements into theLCD driver matrix [3]. In combination with collocated, thin,optical capture and display elements, such as those providedby a SIP LCD, this work suggests a straightforward route toachieving a low-cost, commercially realizable, real-time, 8Ddisplay. The 40in diagonal Samsung SUR40 Sensor-In-Pixeldisplay has 1920 × 1080 resolution, yielding a pixel pitch of55dpi. One goal of this work is to inspire manufacturers toincrease these numbers.

CONCLUSIONWe have presented a glasses-free 3D display capable of react-ing to real-world environmental and user-controlled lightingin real-time. This work paves the way to creating displaysthat can produce physically convincing illusions that partici-pate optically in the environment in which they are rendered.It is our hope that this work will inspire follow-on investiga-tions in the HCI community to more fully explore the greatpotential of 8-dimensional displays.

REFERENCES1. Adelson, E. On seeing stuff: The perception of materials

by humans and machines. In Proc. of the SPIE,vol. 4299, Citeseer (2001), 1–12.

2. Anderson, R., Knowles, W., and Culley, J. Methods andapparatus involving light pen interaction with a real timedisplay, Apr. 12 1977. US Patent 4,017,680.

3. Brown, C., Kato, H., Maeda, K., and Hadwen, B. Acontinuous-grain silicon-system lcd with optical inputfunction. Solid-State Circuits, IEEE Journal of 42, 12(dec. 2007), 2904 –2912.

4. Cossairt, O., Nayar, S., and Ramamoorthi, R. Light fieldtransfer: global illumination between real and syntheticobjects. In ACM Trans. on Grap., vol. 27, ACM (2008),57.

5. Fleming, R., Dror, R., and Adelson, E. Real-worldillumination and the perception of surface reflectanceproperties. Journal of Vision 3, 5 (2003).

6. Follmer, S., Johnson, M., Adelson, E., and Ishii, H.deform: An interactive malleable surface for capturing2.5d arbitrary objects, tools and touch. In Proc. 24thACM symp. on UIST, ACM (2011), 527–536.

7. Fuchs, M., Raskar, R., Seidel, H.-P., and Lensch, H.P. A. Towards passive 6d reflectance field displays. ACMTrans. Graph. 27, 3 (Aug. 2008), 58:1–58:8.

8. Hirsch, M., Lanman, D., Holtzman, H., and Raskar, R.Bidi screen: A thin, depth-sensing lcd for 3d interactionusing light fields. In ACM Trans. Graph., vol. 28, ACM(2009), 159.

9. Hullin, M. B., Lensch, H. P. A., Raskar, R., Seidel,H.-P., and Ihrke, I. Dynamic display of brdfs. ComputerGraphics Forum 30, 2 (2011), 475–483.

10. Hutley, M., Hunt, R., Stevens, R., and Savander, P. Themoire magnifier. Pure and Applied Optics: Journal ofthe European Optical Society Part A 3, 2 (1999), 133.

11. Ives, F. E. Parallax stereogram and process of makingsame. U.S. Patent 725,567, 1903.

12. Izadi, S., Hodges, S., Taylor, S., Rosenfeld, D., Villar,N., Butler, A., and Westhues, J. Going beyond thedisplay: a surface technology with an electronicallyswitchable diffuser. In Proc. 21st ACM symp. on UIST,ACM (2008), 269–278.

13. Lanman, D., and Hirsch, M. Build your own glasses-free3d display. In ACM SIGGRAPH 2011 Courses, ACM(2011), 4.

14. Leitner, J., Powell, J., Brandl, P., Seifried, T., Haller, M.,Dorray, B., and To, P. Flux: A tilting multi-touch andpen based surface. In Proc. 27th intl. conf. extendedabstracts on Human factors in computing systems, ACM(2009), 3211–3216.

15. Levoy, M., and Hanrahan, P. Light field rendering. InProc. 23rd conf. on Computer Graphics and InteractiveTechniques, ACM (1996), 31–42.

16. Lippmann, G. Epreuves reversibles donnant la sensationdu relief. Journal of Physics 7, 4 (1908), 821–825.

17. Nayar, S., Belhumeur, P., and Boult, T. Lightingsensitive display. ACM Trans. on Grap. 23, 4 (2004),963–979.

18. Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M.,and Hanrahan, P. Light field photography with ahand-held plenoptic camera. Computer ScienceTechnical Report 2 (2005).

19. Sato, T., Mamiya, H., Koike, H., and Fukuchi, K.Photoelastictouch: Transparent rubbery tangibleinterface using an lcd and photoelasticity. In Proc. 22ndACM symp. on UIST, ACM (2009), 43–50.

20. Tompkin, J., Muff, S., Jakuschevskij, S., McCann, J.,Kautz, J., Alexa, M., and Matusik, W. Interactive lightfield painting. In ACM SIGGRAPH 2012 E. Tech., ACM(2012), 12.

21. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A.,and Tumblin, J. Dappled photography: Mask enhancedcameras for heterodyned light fields and coded aperturerefocusing. ACM Trans. Graph. 26, 3 (2007), 69.

22. Wetzstein, G., Lanman, D., Hirsch, M., and Raskar, R.Tensor displays: Compressive light field synthesis usingmultilayer displays with directional backlighting. ACMTrans. Graph. 31, 4 (2012), 80.

23. Zwicker, M., Matusik, W., Durand, F., and Pfister, H.Antialiasing for automultiscopic 3D displays. In EGSR(2006).