CubeMap360: Interactive Global Illumination forAugmented Reality in Dynamic Environment

A’aeshah Alhakamy∗, Mihran Tuceryan†∗†Computer and Information Science Department

Indiana University - Purdue University Indianapolis (IUPUI)Indianapolis, USA

Email: ∗aalhakam@iupui.edu;@iu.edu;@purdue.edu †tuceryan@iu.edu; mtucerya@purdue.edu∗Computer and Information Technology School

University of TabukTabuk, Saudi Arabia

Abstract—Most augmented/mixed reality (AR/MR) applica-tions still have shortcomings when integrating virtual objects cor-rectly in real scenes such that virtual objects are indistinguishablefrom reality. While the entertainment industry has accomplishedincredible results in every known form of media, the mainprocess is usually achieved offline. An interactive extraction ofillumination information from the physical scene used to rendervirtual objects would provide a realistic outcome for AR/MRapplications. In this paper, we present a system that capturesthe existing physical scene illumination, then apply it to therendering of virtual objects added to a scene for augmentedreality applications. The illumination model in the developedsystem addresses some components that work simultaneously atthe run time. The main component estimates the direction of theincident light (direct illumination) of the real-world scene througha 360◦ live camera attached to AR devices using computer visiontechniques. A complementary component simulates the reflectedlight (indirect illumination) from surfaces in the rendering ofvirtual objects using cube maps textures. Then, the shadingmaterials of each virtual object are defined which provide theproper lighting and shadowing effects including the two previouscomponents. In order to ensure that our system provides a validresult, we evaluate the error incurred between the real and thevirtual illuminations produced in the final scene based on theshadow allocation which is emphasized while conducting a userstudy.

Index Terms—Direct Illumination, Indirect Illumination, Aug-mented Reality, Image-Based Lighting, Incident Light, ReflectedLight.

I. INTRODUCTION

In rendering virtual objects as realistically as real objectsin AR/MR scenes, it is essential to have correct illumina-tion information. Several factors are involved in presentinga photorealistic, dynamic and interactive augmented/mixedreality scene that seamlessly integrates the various illuminationeffects to show a visually consistent result. In this paper,we explore and utilize algorithms to achieve such a visuallycohesive rendering of the scene using the information extractedfrom the scene about the physical lighting conditions.

A precise HDR light capture technique was developed foroffline rendering outdoor and indoor scenes, which producedaccurate lighting and cast shadows [1], [2], but cannot be

978-1-7281-0137-8/19/ $31.00 ©2019 IEEE

applied in real-time due to the lengthy capture process. There-fore, when developing similar lighting estimations that workin almost real-time, the capturing process must be constrainedto LDR or limited HDR. This cannot produce exact lightingbecause information in highlight areas is lost. To address thisproblem, it is possible to investigate environment map lightingin several components: incident light, ambient reflected light,and shading properties.

The system presented here describes these main componentswhich work simultaneously to accomplish a realistic lookingAR/MR result. The system would be applicable to any ARinput or output device (e.g., a handheld mobile device, a headmounted display or webcam camera) instrumented with a 360◦

panoramic live-feed camera.The main component is the angle estimation of the direct

illumination which is the light that falls from the source ontothe objects directly then reflected towards an observer. In thisstep, we employ a 360◦ live feed camera to capture an image ofthe whole physical environment which is then used to estimatethe light direction.

Another component includes the indirect light which isreflected from object surfaces (of real or virtual objects) thatthen illuminates surfaces of another object (see Fig. 1). Thereflected light constructs a cube map texture from the 360◦

camera point of view looking at the entire environment.The last component applies shading properties for each

virtual object through an OpenGL shading language to expressthe correct lighting conditions and shadowing effects basedon the previous concepts. The purpose of this research is toexplore the integration of these methods and techniques toachieve a realistic augmented reality scene.

An accuracy evaluation of the resulting scenes througherror estimating in our experiments was applied where it wasdetermined that a range of improvement can be inserted forfuture work. The following section discusses some usefulconcepts and past work which inspired this research.

II. RELATED WORK

Rendering a realistic visual object is not a new topic, asthe concept of a believable perception in computer graphics

Fig. 1. Illustration for the difference between the incident light (direct illu-mination) and the reflected light (indirect illumination) and their interactionswith the real and virtual objects.

has been discussed in earlier work. In this paper, we focus onaugmented reality where the virtual objects interact with thephysical scene. This part covers some related and prior work.

First, consider the techniques used to estimate or detect thedirection of incident light (direct illumination) . One of thesetechniques is the dynamic sampling of incident radiance ofcreating a set of samples at the objects center point. However,for more accurate and optimized results that can handle thelocal variation of light over the object, a uniform distributed setof points is employed to sample over the direct illuminationusing the Iterated Closest Point (ICP) algorithm at run-time[3]. The point sampling of image surfaces and ReflectanceProfiles (RP) is also used to detect the incident lights [4]. Theoutcome of their RP based classification matched the initialassumption, so the light source impacted the points which wereclassified as variable RP.

Another proposed method is a hardware solution that wouldcapture the incident lighting through panoramic HDR imagesequences using an optical imaging system that includes boththe reflection and refraction of light utilizing an HDR cameraand a mirror sphere at run-time also known as a light probe [5].Many others have developed algorithms using a sphere modelat the beginning which is then extended to different shapes,observing the Lambertian reflectance properties on the objectswhose sizes were previously known [6].

Although there are no surfaces that are perfect reflectors,the notion of determining the fraction of direct illuminationreflected from a surface has been used previously to take ad-vantage of the Fresnel term and the Schlick approximation [7].The Lambertian model is not a strange concept, where thefraction of incident light is represented by the constant albedoof the surface [8].

Some solutions for spatially varying indoor light have beenproposed where the light source is assumed to be distant fromthe location of the camera, yet the virtual objects can be placedat different spots in the final scene [9], [10].

A light-based model using a light probe is still used inmany applications even to date. In this model, a radiance

map is acquired with full dynamic omnidirectional range closeto the location of the virtual object in the real scene [11],while others [12] have used a fish-eye lens to acquire ahigher resolution that also exhibits specular reflections. AStudierstube Tracker is added in the scene to track the cameraand lamp locations [13].

The environment map is captured with a 180◦ wide-anglefish-eye camera and the image is sampled for light sourcedetection using a variance-cut algorithm which minimizes themaximum variance in multiple divided regions to determinethe brightest area of the image [14] .

The Bidirectional Reflection Distribution Function (BRDF)which is part of the rendering equation is used to estimate thedirection of the light source among the diffuse or glossy partof the objects [15]. Moreover, the inverse rendering problem issolved by factoring the lit-color texture into two parts, directlight and albedo colors [16].

The techniques used to obtain the reflected light, andsimulate the effect of reflected light include, for instance, theenvironment map which is still achieving a plausible outcomein this area. The input image acquired from a distant lightprobe is copied and then flipped horizontally, then before beingsegmented into six faces of cube maps and finally combinedthem to provide a realistic reflected light [15].

The radiance samples are captured in the dynamic scene byrendering six images from each point which is correspondingto the spherical parameterization of the cube maps six faces onthe graphics hardware for precomputed radiance transfer [3].The Spherical harmonic (SH) is also compared to evaluatethe light estimation directly on a cube map which has aunique direction of the unit sphere that is represented by eachpixel [17]. The recorded radiance values are stored to atlastexels to capture the environmental illumination [18].

The utilization of a 360◦ panoramic video is a familiarconcept in virtual reality (VR), where some have used the360◦ pre-recorded video to provide an immersive system andrealistic illumination for a dynamic environment. A VR/MRsystem composes virtual objects into a 360◦-video is presentedin [19] and is used to detect the light source to irradiate theobjects.

Most of the previous work assumed or restricted the envi-ronmental condition and the devices setup, while our approachhas minimal requirements.

III. METHOD

As mentioned, the system tackles realism and visual coher-ence in several components which work simultaneously. A fulland detailed overview of the system is presented in (Fig. 2).

Incident lights (direct illumination). We use the 360◦

panoramic live-feed camera to capture the whole real scene.Then, computer vision and pattern recognition methods areused to estimate and sample the areas of light in the real scene.Then, from the viewpoint of the rendering camera, we createvirtual lights in the virtual scene that mimics the real light forrendering the virtual objects.

Fig. 2. An overview of the whole system including the three components: incident light detection, reflected light simulation and shading materials, in additionto the rendering and interaction process.

Reflected light (indirect illumination). The 360◦

panoramic live-feed camera is used to create a cube map [3]of the whole environment and then reflect it on the virtualobjects. The ability to change the angle and transformation ofthe reflection from multiple objects (real/virtual) is maintainedas if they are inside the physical world space.

Shading materials. Provide the virtual object with everycharacteristic possible to make it look as realistic as the realobjects. Some required passes are performed to reflect themain light, additional lights, and shadows, followed by light-ing mode, texture material, normal map, diffuse percentage,specular map, tangent map, and ambient light. The Image-based mode of the cube maps is included with HDR exposure,reflection percentage and details, refraction percentage andindex, Fresnel width, and BRDF-Anisotropy. Some virtualobjects might not require all of these effects based on theirnormal material in the real world.

Rendering. Is employed to create a more realistic appear-ance, where the global illumination is used in the whole ARsystem. Each object is rendered with the forward renderingpath technique in one or more passes based on the lightsaffecting the object. The base pass is applied for one perpixel main light and all per vertex or Spherical Harmonic (SH)lights. Additional passes render any other per pixel lights byhaving one pass for each light which is treated differentlybased on their setting and intensity. The unity engine isused to take full advantage of parallel support on GPUs inthe shader rendering. The engine provides full resolution fortexture quality, forced on anisotropic textures, with doublemultisampling for anti-aliasing and soft particles.

Tracking. for augmented reality systems, is essential forproviding 3D virtual objects with their locations and orien-tations. In our experiments, an image marker-based trackingtechnique [20] is used as an anchor for the initial locationprovided by the Vuforia Engine. It supports multiple opticaland video see-through stereo rendering displays when therequired VR SDK is installed. However, additional scripts arewritten to maintain the light conditions even with the virtualobject, camera and marker movements.

Hardware Description. The system runs on a PC withan Intel®CoreTM i7-3930k CPU @ 3.20GHz 3201 MHz, sixcore(s), 64.0 GB RAM, and NVIDIA GeForce GTX 970 GPU.For our preliminary result, two cameras are employed forvideo input and environment map which are: a DSLR NikonD7200 and RICOH THETA S 360◦ with the live feed.

Furthermore, the broadcasting software and system registryfor both cameras live feed are required for continuous datainput through the 3D game engine. Thus, the compatibility ofthe devices must be considered while transferring the systemonto different display options.

The following sections explain each component of thesystem in more detail.

A. Incident Light Detection

The system assumes that the main light sources with anarbitrary number of lights and directions can be sampled

through a 360◦ panoramic view. The live feed from two/morefish-eye camera for the current environment provides enoughdata to capture the light area. An OpenCV library is exploitedto sample the light area under certain condition. The methoduses the image frame captured from the camera. The numberof frames updated per second can be controlled to maintain areasonable amount of performance latency. For input readingcompatibility between the 3D engine and computer visionlibraries, the original 2D texture of the video feed (i.e., avideo frame image) is converted into an OpenCV Mat objectwhich then converts the 3D vector in an OpenGL manner usingparallel computation support. After processing each imageframe, a reversed procedure is applied.

An image frame is captured from the live video feed forthe incident light sampling process. The light sources aredetected by identifying saturated bright areas in the imageusing histogram-based thresholding which is unique for eachenvironment’s overall lighting conditions, after appropriatenoise reduction.

A series of erosions and dilations are performed to removeany small blobs of noise. The main goal is to extract thecontours that represent several light areas then sort them basedon the sizes of these areas. The largest area should be assignedto the main light followed by the secondary sources of lightand so on. The image moments of each contour area are crucialto finding the light location for assignment to an identicalvirtual light source (see Fig. 3 ).

Fig. 3. Light source detection of the physical world captured from the360 panoramic view, the environments (a) and (b) has two different lightingconditions where histogram-based thresholding is used to determine the lightintensity sufficiently.

After detecting the light source position in the panoramicand stitched views, the screen coordinates (x, y) are trans-formed to the spherical coordinates (r, θ, φ) using inversespherical projection in terms of Cartesian coordinates (x, y, z).

The ideal location of the light on the spherical view of theenvironment map is going to work as a skybox around ourvirtual object. The Texel to sphere conversion starts with thenormalization of each axis then the spherical coordinates willbe represented as:

Radius(r) =√x2 + y2, θ = tan−1 y

x, φ = cos−1 z

r(1)

The transformation to Cartesian coordinates is affected bythe width and height of the panoramic view and the 360◦

aspect. The concept of equirectangular projection is used inthis work to estimate the location and angle of the light source(see Fig. 4).

Fig. 4. The equirectangular projection from 360◦ panoramic video to sphere.

The panoramic image splits into two sides that represent thelocation as follows:

dw =(360× 2− 1)(x− 0)

(w − 0) + 1(2)

dh =(360− 0)(x− 0)

(w − 0) + 1)(3)

If the light is located on the west side of the entire image,then virtual light corresponding to the real light is representedas the following coordinate.

x =sinφ cos θ

π × dh, y =

cosφ

π × dw, z =

sinφ cos θ

π × 90(4)

On the other hand, if the real light is on the east side of theentire view, then the location is represented as:

x =sinφ cos θ

π × (dh− 90), y = − cosφ

π × (dw − 180), z =

sinφ cos θ

π × 90(5)

The previous steps can be summarized as seen in Algo-rithms 1.

B. Reflected Light Simulation

The reflected light embodies the surrounding atmospherewhere any object in the scene must be affected by it to achievebelievable realism. In this section, the 360◦ panoramic videois also used to create a cube map that will be added onto theobject shading characteristic. The number of details and HDRexposure can be changed based on the environmental intensityand the object material.

The panoramic live stream is converted to cube maps,where Coroutine is deployed to accelerate the construction at

Algorithm 1 Incident Light DetectionGrayscale image.Histogram.Median of the histogram.Gaussian blur.Threshold range by the histogram value to 255.Morph shape (erode and dilate).Find contours.Sort contours based on the area size.Find the center of each contour.Find the mean color (light color) in each contour.Enable a virtual light in the center of the main contours.Rotate and locate the light position based on the transformationfrom 2D Texel to 3D spherical point.Calculate light intensity.Calculate shadow strength.

runtime. Thus, the textures are updated at periodic intervals(10 frames) while the whole code runs at 60 frames/sec. Thespherical panoramas combine a vertical 180◦ angle viewingand a horizontal 360◦ angle viewing. Data about the light arecontained from all directions in these panoramas which can bevisualized on a sphere as comprising points. The cubic formatalso known as cube maps has six faces to fill the entire spherearound the viewer. The maps are created by live video feedfrom the 360◦ camera giving a left, back, right, front, top,and bottom textures. The six faces have appropriate texturemaps whose area is typically arranged in a horizontal crossconfiguration forming an unfolded cube. When these texturemaps are folded, the view is remapped to the cube faces thatfit perfectly (See Fig. 5).

Fig. 5. Cube map faces, an illustration of how the light is going to reflecton the virtual objects.

For this component, the cube map algorithms are imple-mented specifically for the current system. The existent onlinetools that provide this feature requires loading a pre-recordedvideo or a single panoramic image which is not what thesystem is meant to achieve. The creation of the cube maptextures from the panoramic video is about making a bigbox and putting it into a vertex buffer without adding texturecoordinates. Then, the cube map is sorted as six separatedtexture files with the appropriate size. In order to set the pixels

with the right color in every point, the spherical projection iscalculated the vector is normalized as following:

θ = tan−1 z

x, φ = cos−1 z, θ+ =

dir × π180

(6)

dx =θ

π, dy =

φ

π(7)

If the values fall outside the width and height of the originaltexture, the following formula is performed:

px = dx× w, py = dy × h (8)

FinalColor =

(px

h− py − 1

)(9)

After that, the edges in the wrap mode are clamped for allthe components to avoid visible seam on our textures. Minoranti-aliasing using inside filtering is applied while the camerarotates in the scene. The final step is to generate the cubemaps by assigning these textures to our OpenGL cube mapvariables.

The resulting cube maps rendered in each virtual objectshare material which has the same property through the shaderas a skybox to reflect the image-based lighting mode whetherit is a regular reflection, refraction, or Fresnel. Each mode andHDR exposer can be manually manipulated as needed whichis considered next.

C. Shading Materials

A virtual object contains vertices, vertex colors, UV dataand normal which is rendered with a material assigned. Thematerial uses a shader with textures, colors and other propertyvalues.The shading process obtains data from the virtual objectmaterials to draw pixels to the screen depending on its CG orGLSL code.

The shader is a program that computes the shading duringthe rendering process: producing the proper level of light,darkness, and color with special effects that help the virtualobject to look more realistic. In our system, two types ofshaders are generated: surface and unlit. The surface shaderreflects the lights and the entire real scene environment, whilethe unlit shader is used for the objects that do not interact withthe lights but can receive shadows.

1) Surface Shader. Every object in the virtual scene hasthree passes: a forward base for the main light, forwardadd for any additional lights, and shadow caster toshow the reflected shadows. It also has multiple featuresthat can be manipulated to improve the realism of thevirtual object such as: main color, main texture, worldnormal from normal map for more details, lighting modevertex or fragment, Lambert diffuse reflection, BlinnPhong specular reflection, Fresnel, ambient light, image-based reflection, projection to texture space, Ashikhmin,Shirley and Premoze BRDF anisotropy. Thus, the fol-lowing properties are defined in the shader to achieve amore realistic outcome:

• Transform the normal map to the world normalwhich takes a 2D texture for each object wherethe normal at a pixel is converted from the colorvalue. Then, compose and normalized the tangent,binormal, and normal (TBN) matrix. After that,define the returned color as property.

• Lambert Diffuse reflection that requires the normaldirection (N), light direction (L), color (c), anddiffuse factor (f) with attenuation (a), is presentedfor multiple lights using the famous formula: d =∑n

i=0 c× f × a×max(0,−→N ·−→L )

• Blinn Phong Specular reflection is a term that cor-responds to the shininess of selective surfaces thattends to show some highlights.

• Image-Based reflection is where the generated cubemap is loaded to mimic the irradiance environmentmap which is then updated with the live video feed.We can also manipulate the number of details andHDR exposure and the reflection factor as needed.

• Image-Based refraction is a phenomenon of bendinga light wave when it enters a new medium.

• Image-Based Fresnel is concerned with the glassand water reflection and refraction where the eye/camera view affects the normal vector (see Fig. 6)

Fig. 6. The effect of Image-Based Fresnel on the glass.

• Ashikhmin, Shirley and Premoze BRDF anisotropyis a function that describes how much light will bereflected from the certain material when light arrivesor falls on it.

2) Unlit Shader. Some AR applications ignore the fallingshadow of the virtual object on the real-world surfaces.In this part, An alpha test is performed to cut-out themain color of the plane that is placed under the virtualobjects as transparent material that only receive theshadows as an indication of the light detection method.

IV. EXPERIMENTAL RESULTS

Multiple scenes are tested in different lighting conditionswith diverse environmental settings (e.g., dim yellow lamplight, white lamp light, direct sunlight, sunlight from thewindow) to evaluate the visual quality and light estimationsystem. All trials were with the dynamic video feed. Most

of our evaluations are based on the shadow observation bycomparing the shadows of the real and virtual objects in thefinal scene. Furthermore, the light falling and reflected on thevirtual objects is a valid indication of the similarity to thereal light which exists in the real environment. The resultscontinued to be valid even when the lights changed, and whenthe virtual objects moved, rotated, and the marker or cameramoved.

Furthermore, the data feedback from 33 college studentsis evaluated where a questionnaire about the final scene ispresented to the subjects (see Fig. 8). Most of the collecteddata are a focus on rating the objects’ realism where par-ticipants answered on a scale from 1 (‘looks synthetic’) to5 (‘looks real’). As many subjects could not differentiatebetween the real and virtual objects, the average of the realobjects that were mistaken as virtual objects are subtractedfrom the original rating. The virtual objects scored valuesabove the average (see Fig.7). Participants were also askedto provide a general opinion on the correctness of the lightand shadow; the lighting on the virtual objects has a gapgreater than 39.01% while the shadow was more believableby a difference of 18% compared with the real objects.

Trial1 Trial2 Trail31

2

3

4

5

2.52

3.323.54

3

3.333.55

2.83

3.343.56

Acc

urac

yR

ange

[1-5

]

Pretzels Calculator Gray Pen

Fig. 7. Rating of the virtual objects realism in comparison to each other.

The feedback from the shadow allocation indicates that thelight detection should provide an insight to calculate the lightangle of the real objects’ shadows manually, then compareit with the angles that the system estimated (the real objectlength was 7.8 cm). The difference margin in degrees wasfound to be insignificant as it is shown in (Table 1). The lightwas concentrated based on the area center in the thresholdwhere the angle computed manually is illustrated in (Fig. 9).

The statistics of these errors can be computed using theroot-mean-square error (RMSE):

RMSE =

√∑3i=1Error

2i

3= 1.38 (10)

The average error is slightly more than a single degreewhich is quite accurate considering the difficulty of selecting

Fig. 8. Experiment results, each case has (1) panoramic 360◦ view, (2) thethreshold mask for the incident light detection, (3) cube map faces for thesurrounding environment for the reflected illumination, (4) the environmentmap as a sphere, (5) final scene.

TABLE ITHE DIFFERENCE (ERROR) BETWEEN THE MEASURED AND

ESTIMATED ANGLES OF THE LIGHT IN DEGREES.

SceneNo.

ShadowLength (s)

Measured(θ)

Estimated(φ)

Error

Trial1 6 -127.56 -126.98 0.58Trial2 15 27.47 26.24 1.23Trial3 18 203.42 203.01 0.41

the center of the light. The absence of significant trends in theerrors suggests that the light estimation model is sufficientlyprecise.

Fig. 9. An illustration of the system environment setup where the light sourcescan be placed anywhere in the scene.

V. DISCUSSIONS AND CONCLUSION

We presented a method to estimate the incident lightsource, simulate the reflected light and provide virtual objectswith increased realism. The algorithm succeeded in deliver-ing an interactive illumination based on shadow assessment.Nevertheless, the problem of illumination and realism inmixed/augmented realities is still a work in progress due toseveral variables that should be addressed but have a limitationin execution. For instance, both input cameras have a latencyfactor, aperture value (Av) and ISO speed that can changethe whole result with a small range of difference. For futurework, the entire tracking system can be improved where someof the depth data that is represented can support the lightdetection and reflection on a certain location using the depthmap. Object or multi-marker tracking would be more tangiblewith the virtual objects. Furthermore, for a more robust systemachieving a complete GPU implementation would improve thewhole performance for different types of devices.

REFERENCES

[1] T. Ritschel, C. Dachsbacher, T. Grosch, and J. Kautz, ”The state ofthe art in interactive global illumination,” in Computer Graphics Forum,2012, vol. 31, no. 1, pp. 160-188: Wiley Online Library.

[2] D. Schmalstieg and T. Hollerer, Augmented reality: principles andpractice. Addison-Wesley Professional, 2016.

[3] P.-P. Sloan, J. Kautz, and J. Snyder, ”Precomputed radiance transfer forreal-time rendering in dynamic, low-frequency lighting environments,”in ACM Transactions on Graphics (TOG), 2002, vol. 21, no. 3, pp.527-536: ACM.

[4] S. Jiddi, P. Robert, and E. Marchand, ”Reflectance and IlluminationEstimation for Realistic Augmentations of Real Scenes,” in Mixedand Augmented Reality (ISMAR-Adjunct), 2016 IEEE InternationalSymposium on, 2016, pp. 244-249: IEEE.

[5] J. Unger, S. Gustavson, and A. Ynnerman, ”Spatially varying imagebased lighting by light probe sequences,” The Visual Computer, vol. 23,no. 7, pp. 453-465, 2007.

[6] Y. Wang and D. Samaras, ”Estimation of multiple directional lightsources for synthesis of augmented reality images,” Graphical Models,vol. 65, no. 4, pp. 185-205, 2003.

[7] S. Rahman and A. Robles-Kelly, ”Estimating Reflectance Parameters,Light Direction, and Shape From a Single Multispectral Image,” IEEETransactions on Computational Imaging, vol. 3, no. 4, pp. 837-852,2017.

[8] D. Samaras and D. Metaxas, ”Incorporating illumination constraints indeformable models for shape from shading and light direction estima-tion,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 25, no. 2, pp. 247-264, 2003.

[9] J. LopezMoreno, E. Garces, S. Hadap, E. Reinhard, and D. Gutierrez,”Multiple light source estimation in a single image,” in ComputerGraphics Forum, 2013, vol. 32, no. 8, pp. 170-182: Wiley Online Library.

[10] J. LopezMoreno, E. Garces, S. Hadap, E. Reinhard, and D. Gutierrez,”Multiple light source estimation in a single image,” in ComputerGraphics Forum, 2013, vol. 32, no. 8, pp. 170-182: Wiley Online Library.

[11] P. Debevec, ”Rendering synthetic objects into real scenes: Bridgingtraditional and image-based graphics with global illumination and highdynamic range photography,” in ACM SIGGRAPH 2008 classes, 2008,p. 32: ACM.

[12] P. Supan, I. Stuppacher, and M. Haller, ”Image Based Shadowing inReal-Time Augmented Reality,” IJVR, vol. 5, no. 3, pp. 1-7, 2006.

[13] M. Knecht, C. Traxler, O. Mattausch, W. Purgathofer, and M. Wimmer,”Differential instant radiosity for mixed reality,” in Mixed and Aug-mented Reality (ISMAR), 2010 9th IEEE International Symposium on,2010, pp. 99-107: IEEE.

[14] D. Kamboj and W. Liu, ”Improved Variance Cut Algorithm for LightSource Estimation in Augmented Reality,” International Journal ofComputer Applications, vol. 85, no. 19, 2014.

[15] T. Schwandt and W. Broll, ”A single camera image based approachfor glossy reflections in mixed reality applications,” in Mixed andAugmented Reality (ISMAR), 2016 IEEE International Symposium on,2016, pp. 37-43: IEEE.

[16] T. Richter-Trummer, D. Kalkofen, J. Park, and D. Schmalstieg, ”Instantmixed reality lighting from casual scanning,” in Mixed and AugmentedReality (ISMAR), 2016 IEEE International Symposium on, 2016, pp.27-36: IEEE.

[17] L. Gruber, T. Richter-Trummer, and D. Schmalstieg, ”Real-time photo-metric registration from arbitrary geometry,” in Mixed and AugmentedReality (ISMAR), 2012 IEEE International Symposium on, 2012, pp.119-128: IEEE.

[18] K. Rohmer, W. Bschel, R. Dachselt, and T. Grosch, ”Interactive Near-Field Illumination for Photorealistic Augmented Reality with VaryingMaterials on Mobile Devices,” IEEE transactions on visualization andcomputer graphics, vol. 21, no. 12, pp. 1349-1362, 2015.

[19] T. Rhee, L. Petikam, B. Allen, and A. Chalmers, ”Mr360: Mixedreality rendering for 360 panoramic videos,” IEEE Transactions onVisualization and Computer Graphics, no. 4, pp. 1379-1388, 2017.

[20] H. Kato and M. Billinghurst, ”Marker tracking and hmd calibration fora video-based augmented reality conferencing system,” in AugmentedReality, 1999.(IWAR’99) Proceedings. 2nd IEEE and ACM InternationalWorkshop on, 1999, pp. 85-94: IEEE.