IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER … · Reality: A Taxonomy, Methods and Patterns Stefanie Zollmann, Tobias Langlotz, Raphael Grasset, Wei Hong Lo, Shohei Mori and

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1

Visualization Techniques in AugmentedReality: A Taxonomy, Methods and Patterns

Stefanie Zollmann, Tobias Langlotz, Raphael Grasset, Wei Hong Lo, Shohei Moriand Holger Regenbrecht,

Abstract—In recent years, the development of Augmented Reality (AR) frameworks made AR application development widelyaccessible to developers without AR expert background. With this development, new application fields for AR are on the rise.This comes with an increased need for visualization techniques that are suitable for a wide range of application areas. It becomesmore important for a wider audience to gain a better understanding of existing AR visualization techniques.Within this work we provide a taxonomy of existing works on visualization techniques in AR. The taxonomy aims to giveresearchers and developers without an in-depth background in Augmented Reality the information to successively applyvisualization techniques in Augmented Reality environments. We also describe required components and methods and analyzecommon patterns.

Index Terms—Visualization, Augmented Reality, Information Visualization, Taxonomy

F

1 INTRODUCTION

LATEST developments in Augmented Reality (AR)work towards a more accessible technology that

is available to end users as well as professionals andopens new fields of applications, such as entertain-ment or advertisement, but also various other profes-sional applications. This allows developers to createAR applications that overlay digital information intothe field of view of users. An example would bedigital instructions for a car repair overlaid directlyonto the view of this car. In particular, AR frameworkslike ARkit1, ARCore2 or the Mixed Reality Toolkit3

make AR experiences more accessible to a wideraudience. These frameworks provide registration andtracking techniques that are adequate for a lot ofuse cases. However, one main challenge remains: theappropriate visualization of content.

Visualization in AR is coming with the challenge ofhow to integrate digital content with our view of thereal world in a combined Augmented Reality view.This challenge is different from the hurdles in tradi-tional visualization techniques where the presentedcontent is well known. In this work, we investigatedifferent ways of how previous research approachedthe problem of integrating (or compositing) digitalcontent with our view of the real world, which chal-

• Stefanie Zollmann, Tobias Langlotz, , Wei Hong Lo, and Holger Regen-brecht are with the Departments of Computer Science and InformationScience, University of Otago, New Zealand.E-mail: [email protected]

• Shohei Mori is with the Graz University of Technology• Raphael Grasset is with PTC

1. https://developer.apple.com/documentation/arkit2. https://developers.google.com/ar/3. https://microsoft.github.io/MixedRealityToolkit-Unity/

Raw Data Focus Data Geometric Data Image

Filtering Mapping Rendering

Fig. 1: Scientific visualization pipeline with its three mainsteps: filtering, mapping, and rendering.

lenges have been addressed, and identify commonpathways. The main aim of this work is to providea better understanding of the general issues for visu-alization techniques in AR. We specifically aim for awider audience and not only towards researchers whoare AR experts.

We will provide information about the special re-quirements for visualization techniques in AR, as wellas give an overview of the commonly used compo-nents and pipelines, common challenges and pitfalls.

1.1 VisualizationVisualization can be described as the process of con-verting abstract data into a visual representation thatis comprehensible by a human observer. The visual-ization process itself is often described step-by-step inone of the various versions of the visualization pipeline[1]. This allows for subdividing visualization methodsinto sub methods and provides a better overview andabstraction of these methods.

Normally, the visualization pipeline defines threemain steps: filtering, mapping, and rendering asshown in Figure 1. In this concept, the first step,Filtering, is a data-to-data mapping that converts rawdata into focus data, for instance by producing a smallersubset of the raw data set. Mapping is the secondstep and uses the focus data to generate geometricinformation. An example are data points that are


Geometric Data

Registration Data

Camera Image

Data Sources Mapping Compositing View

X

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

AR view

Fig. 2: AR Visualization Pipeline illustrating a naıve overlay using predefined geometric data, a camera image andregistration data for creating the compositing.

mapped to 2D points or a line with a specific color.The last step is the Rendering of this geometric data toproduce a 2D image that can be display on an outputdevice. It is important to highlight here that renderingis seen as one step of the visualisation process. Tokeep to the scope of the paper we will not coverany techniques that are specific to the rendering step,such as photo-realistic rendering. This is a commonseparation, e.g. [2] stating ”Visualization: researchinto methods that use AR to make complex2D/3Ddata easier to navigate through and understand.” vs.”Rendering: research into techniques for computergraphics rendering; and other sensory modalities.”

1.2 Visualization in ARIn contrast to visualization in Scientific Visualizationand Information Visualization, visualization in AR isusually defined in a different way. Available defini-tions focus more on the fact that not only virtualdata is mapped to a visual representation, but alsoon spatial relationships and interaction between thephysical world and raw (virtual) data [3], and howthey are combined to generate the final 2D image.Generally speaking, an important aspect of visualiza-tion in AR comes from Azuma’s definition of AR [4]:the combination of real and virtual information.

This means if we want to use the traditional visu-alization pipeline in the context of AR visualizations,it has to be modified to reflect this combination ofreal and virtual information. Azuma amended hisoriginal survey by emphasizing how AR systems needto have a good registration process to align virtualand physical objects well [5]. By adding registrationinformation, a camera image (representing the envi-ronment in video see-through AR interfaces and forextracting environment information captured in theimage) and a dedicated compositing step to the originalpipeline, we can adapt it to reflect the characteris-tics of AR visualization in a naıve AR visualization

pipeline (Figure 2). In particular, here it is important tomention that the original rendering step is replaced bya compositing step that addresses the need to combinedata from different sources relevant to the AR visual-ization. Overall, we can say that visualization in ARdiffers from the traditional definitions of visualizationbecause it is a combination of the data to displaywith information that is part of the actual physicalenvironment of a user.

It should be noted at this point that this workintentionally excludes issues and challenges that areunique to specific AR displays (e.g Spatial AR, videosee-through displays, optical see-through displays).Realizing an AR interface for example using anoptical-see through display will come with its ownchallenges because of the characteristics of the usu-ally half-transparent display [6], [7], [8]. A generaloverview of some of these display specific issues canbe found in other works [9], [10].

1.3 ChallengesAt first glance, the implementation of the additionalrequirements that come with the combination of vir-tual data and the physical environment of the userseems to be straight-forward if the registration be-tween virtual content and the physical world repre-sentation is known (for instance in terms of a cam-era transformation matrix). In this case, data can becombined by simply overlaying the registered virtualcontent to the user’s view. However, in a lot of situ-ations, a compositing using such a naıve overlay canlead to serious perceptual problems that may preventthe user from comprehending the visualization.

For instance, one of the problems that often arisesin AR when visualizing virtual objects using a naıveoverlay is missing visual coherence [9]. In this case,the virtual content is not coherently integrated intothe AR view. This can happen when perceptual cuesare interfering with each other or are missing. This is


Fig. 3: Visualization problems caused by naıve compositing functions. Left) Virtual planned lamps are perceived to belocated in front of the house not next to it. Middle) Virtual pipes seem to float above the ground. Right) In this touristscenario the naıve overlay of sight labels produces information clutter.

a major challenge, since users need these perceptualcues to understand the spatial relationships betweenvirtual data and real world. If, for example, the com-positing method does not take occlusions betweenvirtual and physical objects into account, it will resultin wrong depth perception. The virtual objects willalways be seen as being in front of the physical worldobjects. Figure 3 (Left) demonstrates this problemwithin an AR planning application. Planned lampsare superimposed on video images to allow users toexperience what it would look like if the lamps werereal. Unfortunately, the missing shadows and occlu-sion cues lead to the wrong perception of floatingvirtual lamps and a lack of coherence.

A similar problem occurs when using a simple over-lay to visualize information that is naturally invisiblesuch as in popular X-Ray visualizations utilizing anAR interface [11]. In Figure 3 (Middle), several subsur-face pipes visualized in a X-Ray view are shown. Sincethe pipes are just naıvely superimposed on the videoimage, there is a lack of natural depth cues. Instead ofbeing perceived as being located subsurface, the pipesseem to float above the ground and making it difficultfor the user to judge where exactly they are located.

As those examples show, AR visualization tech-niques have to address these challenges that arisefrom combining virtual and real data, objects, andenvironments.

1.4 ContributionThe main aim of this work is to provide an overviewof important components for AR visualization, exist-ing techniques and discuss common pattern. For thispurpose, we investigated existing AR visualizationtechniques. We explain their components and lookinto ways how to classify and group existing methods.While in Scientific Visualization and Information Visual-ization it is common to use the Visualization Pipelinefor mapping the different steps of a visualizationtechnique, in AR these traditional pipelines are notworking because of the amount and different types ofinputs (such as the digital data, and the information

from the physical environment) that are involved inan AR visualization. Overall, the main contributionsof this work are

• the identification of important components forrealizing an AR visualization,

• the visualization pipelines adapted to the specialneeds of AR,

• a taxonomy that structures existing visualizationtechniques,

• and the identification of common patterns(pipelines).

2 RELATED WORK

While visualization and interaction techniques forvirtual environments have been researched inten-sively [12], [13], [14], [15], there is only limited relatedwork that classifies and analyses visualization tech-niques for AR. There has been some previous workon surveys and literature review of AR in general[16], [2], [17], [18]. These works provide surveys ofpapers and topics in the field of AR in the last 15 to50 years. There are also surveys published for morespecific topics such as applications of AR [19], orperceptual problems in AR and visual coherence [10],[9]. However, for the topic of visualization in AR, onlyminimal work is available.

Willet et al. defined general concepts of data rep-resentation and presentation for AR and looked intorelationships between data, physical environments,and data presentations [20].

In 2011, Kalkofen et al. provided an overview ofAR visualization techniques [3]. Their work providesa comprehensive overview of existing visualizationtechniques and identified main topics such data inte-gration, scene manipulation and context driven visu-alization. While their work provides a good overviewof visualization techniques for AR, so far there isno investigation of common patterns, pipelines orclassification of AR visualization techniques. In thiswork we want to bridge this gap and provide acloser look into existing pipelines to identify patterns,similarities and challenges specific to AR while also


integrating many newer works that have not beenconsidered by Kalkofen et al..

3 DATA COLLECTION

To gather research within the field of visualizationmethods in Augmented Reality, we used a definedset of search criteria. We made use of the two maindatabases within the field (the IEEE Xplore database,and the ACM library).

We used the following paper gathering strategy:1) Find Augmented Reality and Visualization in title2) Find Augmented Reality in keywords and Visual-

ization in title3) Find Augmented Reality and Visualization in au-

thors keywords.This search resulted in a total number of 429 dif-

ferent works. From these works, we removed allthose works that were not introducing an AR visu-alization technique (e.g. works that were using ARvisualization for a specific application scenario orfocusing on interaction techniques, as well as worksthat were purely on user studies, work on virtualreality, republished content, as well as non-Englishworks, Awards, Table-of-Contents, Keynotes, Demoabstracts). After filtering we ended up with roughly40 works that potentially present new visualizationtechniques. From there we filtered even more strictremoving works that have no visualization techniquespecific to AR, for instance such as works that areapplying standard information visualization in ARwithout any specifics to AR (e.g. [21], in total 13).After this initial data collection we also queried forpapers combining Augmented Reality and Rendering.Despite the more than 1000 results this query addedonly two additional papers within the scope of thiswork. Finally, given our expertise in the field weadded missing important related works that werelacking keywords in the databases (e.g. many olderworks) and double-checked for references within theidentified papers. In total we ended up with 67 worksthat present new AR visualization techniques.

4 AR VISUALIZATION COMPONENTS

Based on the information from the gathered researchworks, we identified several reoccurring componentsthat were used by most of those works. These com-ponents include the camera image, registration data,geometric data, and often some form of masking data.

Camera Image (I): A camera is an essential com-ponent in most AR systems independent of the usedAR display (e.g. video see-through or optical see-through). The camera image captures the real en-vironment around the user and is often used forvisual tracking. Moreover, the camera image is essen-tial when using a video see-through (VST) approachwhere the real environment is shown to the user as a

live video feed with minimal latency. However, alsowhen using optical see-through displays the cameraimage provides many important information for ex-ample when extracting saliency information or otherimage cues.

Registration Data (R): Registration data plays amajor role by supporting the alignment of virtual andphysical objects. Registration in the AR context is thespatial relation between the virtual objects and thereal environment [22]. Registration data can be pro-vided by different means, e.g. six-degrees-of-freedomtracking systems, AR fiducial markers [23], vision-based tracking [24], [25], [26], image recognition-baseddetection or sensor-fusion supported geo-referencedregistration data obtained (e.g. [27]).

Geometric Data (G): Geometric data is the datathat is not physically present within the user’s envi-ronment. It ranges from 2D data such as labels, over2.5D data such as billboards to 3D data such as 3Dmodels. It will be the task of the application to usethe relevant registration data to ensure the geometricdata is properly displayed with high level of accuracyin terms of alignment.

Context Data (C): In addition to the geometric datathat we want to visualize, data that represents aspectsof the physical environment is considered as contextdata. Context data is an important component as itsupports the combination of virtual data with thephysical environment.

For instance, this could be a phantom model that isused for creating correct occlusions in 3D space or acontext mask (image mask) that represents importantareas of the physical environment and that is used tohighlight a certain area in the camera-image space.

For context data it is important to note that it canbe a static representation (e.g. by a CAD model) ora dynamic representation that continuously updatesthe representation of the physical environment (e.g.RGBD data stream or a point cloud) and as suchcontext data is related to context data that can beobtained from sensors in the environment and whichis also increasingly used in AR for context awareness[28].

Coordinate systems: It is important to note thatall the described components come with their owncoordinate reference systems. These coordinate sys-tems can be , model-centred, world-centred, sensor-centred, user-centred, camera-centred. For instance,the geometric data may be defined with a model-centred space, the physical representation data maybe coordinated with a capture devices such as aKinect, the camera image is defined as 2D image andcomes with its camera centred coordinate system andthe registration data could be aligned to a trackingdevice. The existence of multiple coordinate systemscomes with its own challenges of mapping betweenthe different coordinate systems. This mapping stepis part of the AR visualization pipeline.


5 AR VISUALIZATION TAXONOMY

In order to identify common pipelines and patternsof visualization techniques in AR, we started creatinga taxonomy of AR visualization techniques. Such aclassification contributes to the understanding of sim-ilarities and differences between different techniquesand will also help to identify common patterns.

Previous work focused on classifying perceptualproblems in AR [10]. However, to our knowledgethere is no work on classifying the characteristicsof visualization techniques in AR to date. Elmquistet al. [15] did a classification within the area of 3Docclusion management for virtual environments. Theydescribed the design space of 3D occlusion manage-ment using a set of identified domains. Based onthe design space, they then proposed a classificationof 3D occlusion management techniques and used itto classify 50 different techniques. They used theirtaxonomy to identify areas that are not covered byexisting techniques. Likewise, the introduction of ataxonomy for visualization techniques in AR can helpus to identify gaps for visualization in AR.

We follow Elmquist’s work by defining the designspace of AR visualization using a set of dimensions.Similar to Elmquist’s work, for each dimension weidentified a set of domains that characterize the di-mension. Each visualization technique is then mappedto a domain within each dimension. This allows us todefine a consistent language and to classify the tech-niques. By analyzing the related work, we identifiedthe following common reoccurring dimensions:

• Purpose - What is the main goal that this visual-ization technique is achieving?

• Virtual Data Visibility - Is the virtual data directlyvisible, or are there any occlusion or X-Ray viewtechniques used?

• Virtual Cues - Is the visualization technique in-tegrating any virtual cues to support the ARvisualization?

• Filtering - Is the data presented unfiltered or isfiltering used to reduce the amount of data?

• Abstraction - Is the raw data visualized or isthere any additional step to provide a degree ofabstraction/changed representation of the inputdata?

• Compositing - How is the combination of realand virtual input achieved?

5.1 Purpose

A common goal of AR visualization techniques isto improve the presentation of virtual content beingintegrated into the physical world. However, thereare different aspects each visualization technique ad-dresses primarily. From the related work, we foundthat those aspects mostly focus on a) achieving visualcoherence, b) a better depth perception, c) reducing

information clutter, d) supporting exploration, and e)directing attention.

a) Visual Coherence - Visual coherence in AR oftenfocuses on a convincing integration of virtual contentinto the physical world. Often, AR applications arelacking of visual coherence due to missing depthcues, such as occlusions or shadows. In order toaddress this problem, researchers in AR proposedvarious techniques that achieve visual coherence byextracting and using natural cues from the physicalenvironment[29], [30], [31], [32], [33].

We are referring to these natural cues as physicalcues, since they can also be found in the physicalworld. These cues try to mimic the natural behaviorof physical objects for virtual objects. The main goal isto achieve visual coherence based on the assumptionthat our visual system knows how to process thesecues.

b) Depth Perception - Depth perception for humansis complicated if the objects do not follow expectedprinciples [34]. This is often the case in AR, it mayhappen when visualizing occluded objects, floatingobjects, or in general objects that are too abstract tohold normal physical characteristics. This is relatedto visual coherence. In order to support the depthperception in AR, additional cues are required. Werefer to these cues as virtual cues, since they arenot naturally available in the physical world. In theliterature, they are also called graphical aides [35]. ARvisualization methods have to integrate these cuesadditionally.

c) Clutter reduction - With the increasing amountof omnipresent information, the presentation of it ismore likely to become subject to clutter. Consequently,researchers in the field of HCI and Information Visu-alization investigate the issue of information clutterfor a long time. In 2005, Rosenholtz et al. provided adefinition of clutter in visualization systems:

”Definition: Clutter is the state in which ex-cess items, or their representation or organi-zation, lead to a degradation of performanceat some task.” [36]

In the research field of Information Visualizationseveral techniques have been developed that aimto reduce information clutter, such as filtering theamount of objects or view distortion techniques thatallow to magnify or rearrange objects of interest.

In AR visualization, complex data is often em-bedded in complex physical environments that arecrowded with information by nature. Thus, informa-tion clutter is a big issue in AR visualization. In orderto address this problem, researchers introduced meth-ods that focus on the problem of information clutter inAR environments. Similar to the methods available forInformation Visualization, research groups proposedmethods that either reduce the amount of informationby filtering the presented content or by using spatial


distortion techniques to rearrange the objects in amore comprehensible way.

d) Exploration - Another objective that has beenaddressed by AR visualization techniques is the ex-ploration of content or environments. There are sev-eral visualization techniques that support the taskof exploration, for instance by providing additionalinformation for comparison [37] or by providing con-textual information for exploring a scene [38].

e) Attention direction - Directing the attention ofusers in a certain way is another ability that ARvisualization techniques can achieve. For instance, bymasking or highlighting certain areas they can directmore attention to a desired region of interest.

Domain: Visual coherence, depth perception, clutterreduction, exploration, attention direction

Example Techniques:Visual coherence [30], [39], [40], [41], [42], [43], [44],

[45], [46], [47], [48], [32], [49], [31], [50], [51], [52], [53],[54], [55], [33], [56], [57],

Supporting depth perception: [58], [59], [60], [61],[62], [63], [64], [65], [35], [66], [67], [68], [69], [70],

Reducing information clutter [71], [72], [73], [74],[75], [76], [77], [66], [78], [79], [80], [81],

Exploration [82], [37], [83], [84], [38], [85], [86], [87],[88], [89], [90], [91]

Attention direction [92], [93], [94], [95], [96].

5.2 Visibility of Virtual Data

The visibility of virtual data has an influence on thevisualization problems that need to be solved. Virtualdata can be completely or partially occluded by phys-ical objects, or be completely visible or even out ofview. This is particularly evident in the visualizationof invisible data, so-called X-Ray visualization [97],[50], [31], [11]. X-Ray views are popular techniquesin AR and are used for various applications such assubsurface visualization [31] or medical applications[47]. This kind of visualization has special challengesin terms of visual coherence, since some of the naturaldepth cues are not available. Thus, the visualizationof occluded and semi-occluded virtual data is stronglyrelated to the primary goal of visual coherence. This isalso reflected in the parallel set visualization of relatedworks where a large amount of works that havethe visual coherence as primary purpose work withoccluded and partially occluded virtual data (Figure5, blue stream). In contrast, for the visualization ofvisible virtual data the visualization purposes aremore diverse covering aspects of depth perception,information clutter, exploration and attention direc-tion. In addition, virtual data can also be out of viewwith visualization techniques focusing on guidance tothese out of view elements.

Domain: Occluded, Partially occluded, Visible, Out-of-view

Techniques:

Occluded: [58], [59], [61], [64], [65], [66], [67], [68],[82], [72], [74], [76], [77], [66], [47], [32], [49], [31], [50],[51], [52], [33], [56]

Partially occluded: [30], [39], [41], [42], [43], [45],[46],

Visible: [92], [93], [94], [95], [96], [60], [62], [63], [35],[69], [70], [37], [83], [38], [85], [86], [87], [90], [91], [71],[73], [75], [78], [79], [80], [81], [40], [48], [53], [54], [55],[57].

Out-of-View: [84], [88], [89].

5.3 Depth CuesAs highlighted by Elmquist et al. depth cues havea major impact on the users’ understanding of theirenvironment [15]. Because of this, Elmquist et al. cap-tured the degree of depth cues within their taxonomyfor 3D Occlusion Management. As shown by previouswork [98], [11] depth cues have also a strong impacton the spatial understanding within AR. In contrastto Elmquist et al.’s work that captured the degree ofdepth cues we decided to capture their characteristicsspecific for AR by using the following three mainoptions: Physical cues, virtual cues or no additionalcues.

a) Physical cues - We define physical cues as cuesthat try to mimic or rebuild natural pictorial depthcues, such as occlusion or shadows. They can be com-puted from different information sources that containcontextual information about the physical and virtualworld. For instance, edges can be extracted from thecamera image and used as natural occlusion cue.

b) Virtual cues - We define virtual cues to begraphical aids that are naturally not available in thephysical world such as virtual scales, measurementsor other graphical hints.

Domain: none, physical, virtualExample Techniques:None: [65], [82], [72], [76], [77], [66], [92], [94], [95],

[96], [37], [83], [38], [85], [86], [87], [91], [71], [73], [75],[78], [79], [80], [81], [40], [48], [53], [54], [55], [57]

Physical: [47], [32], [49], [31], [50], [51], [52], [33],[56], [30], [39], [41], [42], [43], [45], [46], [90]

Virtual: [58], [59], [61], [64], [66], [67], [68], [74], [84],[88], [89], [93], [60], [62], [63], [35], [69], [99], [70]

5.4 AbstractionAbstraction in a visualization context allows for thereduction of visual complexity by finding differentrepresentations while preserving only the relevantdata. This definition is close to the one of Strothotte[100] who define abstraction as the process in whichcomplex information is refined to signify the importance ofcertain features from the underlying model to provide bettercontext and visualization. Abstractions allow us to re-duce the amount of information showed by mappingor visualizing through a wide range of alternativerepresentations. We differentiate between techniques


that display the concrete data and techniques thatmodify the visualization using an abstraction of thedata. In AR one of the challenges is to find a suitableamount of abstraction that is still coherent with a notabstracted version of the real environment.

Domain: Concrete , AbstractTechniques:Abstract: [96], [78], [79], [80], [70], [95], [83], [86],

[87], [84], [88], [89], [60], [69].No abstraction: [65], [72], [66], [74], [82], [76], [77],

[92], [94], [37], [38], [85], [91], [71], [73], [75], [81], [40],[48], [53], [54], [55], [57], [47], [32], [49], [31], [50], [51],[52], [33], [56], [30], [39], [41], [42], [43], [45], [46], [90],[58], [59], [61], [64], [66], [67], [68], [93], [62], [63], [35].

5.5 FilteringSimple AR visualization methods, usually apply nofiltering and render a predefined geometric model ina naıve overlay. When it comes to the visualization ofmore complex data, an adequate filtering is requiredto avoid information clutter. In particular, informationclutter is increased in outdoor environments due tothe complexity of the physical environment. We candivide the dimension into techniques that use rawdata and techniques that apply filtering.

Domain: Raw, FilteredTechniques:Raw: [82], [76], [77], [92], [94], [95], [37], [83], [38],

[85], [86], [87], [91], [71], [73], [75], [81], [40], [48], [53],[54], [55], [57], [47], [32], [49], [31], [50], [51], [52], [33],[56], [30], [39], [41], [42], [43], [45], [46], [90], [58], [59],[61], [64], [66], [67], [68], [84], [88], [89], [93], [60], [62],[63], [35], [69].

Filtered: [65], [72], [66], [96], [78], [79], [80], [74], [70].

5.6 CompositingThe way the virtual and the physical information iscomposed into the final AR view depends on the ARdisplay (e.g., optical see-through, video see-through,spatial AR) but also depends on the objectives ofthe visualization. As discussed in the beginning, thispaper focuses on the visualisation technique and triesto abstract the AR display as much as possible. How-ever, we acknowledge that many existing techniqueswere presented within video see-through systems andsome compositions are harder to realise in opticalsee-through systems as parts of the compositing arehappening in the optical combiner. With respect tothe visualisation, we differentiate between techniquesthat use a simple overlay, blending (or masking if thecontext mask is binary), information transformation,or inpainting.

a) Overlay - Overlay describes all techniques wherethe virtual geometric data G is simply superimposedon the physical world representation. For simplicity,we discuss this here in the context of a video see-through display where the physical world is captured

by the camera image I . When considering optical see-through devices much of the compositing is happen-ing in the optical combiner where I is basically theenvironment light representing the physical world.The combination of the environment I and virtualcontent G defines the final compositing O = G+ I .

d) Blending - Instead of simply combining virtualcontent and real content, blending techniques use ablending parameter to control how virtual and physi-cal content is combined. They use a blending mask (orcontext mask) M to control which virtual and whichphysical information is visualized or omitted andto which amount. The mask information is usuallyextracted from the context data (C) The final com-positing is then given by O = MG+ (1−M)I . WhenM is represented by binary values the compositing is amasking, when values range between 0-1, the mask-ing process becomes a blending. Blending combinescontent by using a weight in the compositing stepand can even be used for achieving a multi-layeredblending by using multiple weighting masks (Mi).

b) Transformation - The transformation (T) of in-formation is a compositing technique that either spa-tially manipulates physical or virtual items or ma-nipulates their appearance for visualization purpose.These techniques create the final compositing by O =TG(G) + TI(I) and depend on the transformation ofvirtual content TG and of physical content or contextinformation TI .

c) Inpainting - Inpainting is the process of syn-thetically reconstructing parts of the video image thatwere lost due to occlusion from undesired objectssuch as markers or robot arms. It often involves amanipulation of the camera image and does not affectthe virtual geometries O = IN(I) +R(G).

Domain: Simple Overlay, Blending(Masking),Transforming Information, Inpainting

Example Techniques:Simple Overlay: [80], [70], [83], [86], [87], [84], [88],

[89], [60], [69], [72], [66], [38], [85], [91], [73], [40], [54],[55], [90], [64], [66], [68], [93], [62], [63], [35],

Masking/Blending: [96], [95], [65], [74], [82], [92],[71], [47], [32], [49], [31], [50], [51], [52], [33], [56], [30],[39], [41], [42], [43], [45], [46], [58], [59], [61], [99]

Transformations: [78], [79], [76], [77], [94], [37], [75],[81], [57], [67]

Inpainting: [53], [48].

5.7 Summary

By investigating the similarities and differences ofvisualization techniques within the related work, weidentified a set of dimension and their domains toclassify existing AR visualization techniques (Figure4). Plotting the visualization techniques sorted alongto those dimensions and domains gives an overviewof the distribution of common pathways and designchoices.


Fig. 4: AR visualization techniques mapped to the taxonomy (limited to publications with citation counts above 30according to Google Scholar, Feb. 2020). Interactive online version available with all publications. [101].

The grouping in Figure 5 of the classification bringsup a few interesting aspects. First, it shows thatvisualization techniques used for occluded and par-tially occluded data in AR are either focusing onvisual coherence or on depth perception. While for thepartially occluded data, the nearly exclusive purposeis to achieve visual coherence (often using physicaldepth cues); for completely occluded data the primaryobjective is more diverse, but also has a strong focuson visual coherence or depth understanding.

We can further see that some dimensions havean equal distribution in their domains, while othersseem to be clustered to one domain. For instance,the usage of physical, virtual and no cues is nearlyequally distributed. In contrast, only a small numberof techniques apply a filtering technique (14%). Withthe continuous growth of available data in general, weassume that aspects of information filtering specific tothe needs of AR will become more important in thefuture.

Another important aspect that becomes evidentfrom our classification is the relationship betweenmain purpose and data visibility, depth cues, filteringas well as compositing. Firstly, it seems that mostvisualization techniques that support depth percep-tion use simple overlays of virtual depth cues andno filtering. Improving the depth perception seemsto be of interest for visible as well as for occludedinformation. Secondly, visualization techniques thataim to support seamless visual coherence are oftenused in the context of occluded and partially visiblevirtual information. In contrast, visual coherence inthe context of visible virtual information has onlybeen explored by a few works. However, we assumethat with an increasing level of fidelity of virtualmodels the need for addressing visual coherence forvisible virtual data will increase.

In order to integrate hidden virtual information in

the compositing, most visualization techniques applya blending method.

Another interesting aspect that our classificationshows is that filtering techniques are not that exclu-sively used for reducing information clutter as onewould expect, they are also used for supporting depthunderstanding and attention direction.

6 DESIGN PATTERNS: AR VISUALIZATIONPIPELINES

Based on the classification in the taxonomy, we iden-tify which visualization techniques often used forwhich visualization purpose and which kind of vi-sualized data. In this section, we use our dimensionalspace to refine the traditional visualization pipelinefor AR that we described in the introduction. OurAR visualization pipeline provides a simplified rep-resentation of the visualization process. To be able torepresent the complete classification of exiting tech-niques, we describe different implementations of theAR pipeline, each addressing different visualizationproblems.

As mentioned before, for simple AR scenes thatcontain no occlusion and no complex data, a simpleoverlay compositing can be used that combines adefined registered virtual geometry with the environ-ment (e.g. as video image in video-see through or viathe optical combiner for optical-see through, Figure 2).Examples for this kind of visualization are the MagicBook where virtual content is overlaid over a book(video see-through) [103] or the visualization withinthe Touring Machine (optical see-through) [104].

The simple pipeline is not working for more com-plex situations with partially or completely hidden, orcomplex information. As identified in the taxonomy,several research groups developed methods that aimto increase the comprehensibility in these situations.


Fig. 5: Grouping of AR visualization techniques in a parallel set visualization. Please note that there is a different orderof domains compared to the text due to grouping reasons to highlight the clustering between different streams Interactiveonline available version [102].

Our taxonomy shows that these objectives areachieved by adding, removing or transforming differ-ent kinds of information. This requires that we adaptthe visualization pipelines for the different needs.

We used the taxonomy also to identify frequentlyused pipelines. These pipelines can be used as designrecommendations for future research.

6.1 Physical Depth Cues PipelineThe strongest stream represented by 18 works is ad-dressing visual coherence using a blending of physicaldepth cues. This design pattern is used mainly foroccluded and partially occluded virtual data. Thephysical depth cues are used as context data and caneither be image-based or model based or a combina-tion of both depending on where they are extractedfrom.

6.1.1 Image-based Physical CuesImage-based techniques achieve visual coherence byextracting physical cues from video images. Theycan be used for creating physical cues in situationswhere the depth order of virtual and physical worldis known (for instance through a semantic meaningas we have it for the visualization of subsurface in-frastructure) and no accurate and precisely registered3D model of the occluding physical world object isavailable (Figure 6 Left). Since such an accurate model

Fig. 6: Using visual saliency extracted from a camera imagefor providing physical depth cues in an X-Ray view of un-derground pipes (left) and a virtual room (right) (Zollmannet al. [11].)

of the physical context may be not available in everyscenario, image-based techniques focus on creatingphysical cues based on 2D physical world data givingby the camera image. In Figure 7, we show the processof extracting physical cues from the camera imageusing the our adapted AR visualization pipeline. TheAR pipeline reflects how important elements fromthe camera image are filtered and mapped to ancontext mask (Figure 7, (Left)). These context masksare then combined with the camera image and virtualgeometries to create the final AR visualization usingblending.

Such an approach has been introduced by Kalkofenet al. [105]. In their work, they extracted edges froma camera image and used them to create edge-basedghostings. In this approach, the edges are rendered ontop of the video image and the virtual content. The AR


Filtering

Raw Data Focus Data Geometric Data

Filtering Mapping

AR view

Registration Data

Camera Image


X

Image-based Cues

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

Mapping

Compositing

Contextual Focus Data

Fig. 7: AR visualization pipeline for extracting image-based physical cues. (Data sources) Important elements from thecamera are extracted and (Mapping) mapped to an importance map that represents the physical cues. (Compositing) Thesecues are then combined with the camera image (for video see-through) and virtual geometries to create the final AR view.

visualization pipeline in Figure 7 reflects this: 1) thecamera image is used to extract edges as contextualfocus data (filtering), 2) the edges are mapped to aghosting mask that is 3) used in the final compositingstep. Bichlmeier et al. extended this approach by usinga combination of edges and bright pixels as physicaldepth cues [106]. Another approach that uses edges asinput to create physical cues is the method of Averyet al. [32]. They applied edges to improve their X-ray vision system in outdoor environments. Basedon this work, Zollmann et al. as well as Sandor etal. later on defined the physical cues as being saliencyinformation [50], [31]. They computed saliency masksfrom the camera image and the layer of virtual contentto decided which information should be preservedin the final rendering (Figure 6). All these methodswork well in situation were enough meaningful datais available in the camera image, but will fail forpoorly textured scenes.

6.1.2 Model-based Physical CuesIf a model representation of the environment is avail-able, this representation can be used to derive pictorialcues. In Figure 8, we depict our AR visualizationpipeline using contextual data to derive cues for thescene integration. Contextual data as additional datasource is only useful, if an accurate 3D registrationand meaningful models are available.

Some approaches use contextual data directly forocclusion culling to provide occlusion cues. One ofthe earliest approaches in this field used an interactivemethod to manually align models of physical worldobjects and applied these models for occlusion culling[30]. Fiala combined a 3D model of the occludingobject and marker tracking for occlusion culling [44].Breen et al. also proposed to use stereo vision to createa 2.5 depth map for occlusion culling [30]. A similarapproach was applied by Fischer et al., who used dy-namic 2D camera and background information [107]

or a time-of-flight camera to create a depth map asinput for occlusion culling, respectively (Figure 9,Left) [108].

More recent approaches use 3D models of the phys-ical environment for increasing the visual coherencein X-Ray AR by deriving physical cues from the geo-metric or visual properties of the model. For instance,Lerotic et al. [109] presented an approach to main-tain salient details of an occluder model from a pq-space-based non-photorealistic rendering. Bichelmeieret al. used ghostings from registered volumetric datato improve depth perception in AR applications incases where hidden structure is of interest [45]. Forthis purpose, they used the curvature, the angle ofincidence and the distance falloff to compute thefinal transparency in the ghosting. Kalkofen et al.demonstrated how to create ghostings based on ananalysis of registered 3D CAD models [105] ((Figure9, Middle)).

These last three model-based approaches for X-Rayvisualization only work well if the models of the oc-cluding object show interesting features in their geom-etry. Mendez and Schmalstieg presented an approachthat allows to create comprehensible ghostings forrather simple shaped and sparsely textured physicalobjects [110]. By mapping a predefined importancemap on the model of the occluding physical object,selected areas of the physical object can be preserved(Figure 9, Right).

However, existing model-based approaches focusnearly exclusively on static environments, since inthis case it is easier to either build a model of theenvironment or capture it.

6.1.3 Combining Model-based and Image-basedPhysical Depth Cues

Another method is to complement the depth cueswith both model-based cues and image-based picto-



Filtering Mapping

AR view

Registration Data

Camera Image


X

Contextual Geometry

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

2.5D & 3D Contextual Data

Contextual Focus Data

Filtering Mapping

Compositing

Fig. 8: Pipeline for creating model-based physical cues. A 2.5D or 3D representation of the physical world is used asadditional input for the visualization to create physical cues. The camera image is only needed for video see-throughdisplays or when extracting other image features.

Fig. 9: Examples for using model-based physical cues. Left) Using aerial 3D reconstruction data for occlusion culling(Zollmann et al. [69]). Middle Left) A ghostings based on edges extracted from a registered 3D CAD model (Kalkofen etal. [105] c©[2009] IEEE). Middle Right) GIS data used for occlusion culling. Right) GIS data applied for occlusion culling(Skinner et al. [70])).

rial cues. Rather than relying on either the model-based cues or pictorial depth cues, a merge of bothcould provide more accurate representation of theenvironment, decreasing the chance of visualizationappearing unintentionally in wrong locations. An ex-ample of a combined method is a depth map estima-tion done by combining sparse 3D models from GISdatabase with the existing images and videos [111].This method uses the segmented images and withthe aid of the sparse models, able to estimate depthand planes which in return is useful for visualizationsuch as annotations, surveying or placement of virtualitems. As mentioned in the previous section, outdoormodel-based cues are of higher difficulty to producedue to the complexity but with a combination of bothmethods it greatly improve performance.

6.2 Virtual Depth Cues Pipeline

The second largest stream (14 works) are visualizationtechniques that use an overlay of additional virtualdepth cues. In this context we can differentiate be-tween methods that use geometrical virtual depthcues or appearance based virtual depth cues. Thesetechniques were used for visible or occluded virtualdata.

6.2.1 Geometrical Depth Cues

Predefined virtual geometries, such as virtual groundplanes or parallel lines, support the depth compre-hension by providing additional depth cues (Figure10). Usually, these additional cues are available ina predefined geometric representation. For instance,Livingston et al. included a set of parallel lines (calledtram lines) into their visualization of colored makersto improve the depth perception in an indoor andoutdoor scenario [35]. Additionally, they added gridpoints to the tram lines (Figure 11, Left). The authorsconducted a user study investigating this visualiza-tion technique and confirmed on a positive effectfor depth estimation outdoors. It seemed that theusers were tending to decrease overestimated depthjudgments in outdoor environments. For indoor usageadding the tram lines was counterproductive, since itdecreased the already underestimated depth.

Livingston et al. also introduced other examplesof using external geometries to improve the depthperception in AR. For instance, they implemented aground grid visualizing a virtual plane on the groundthat either shows the distance to the user with concen-tric circles or with parallel lines [66]. Their graphicalaid restore the visual cues of height in visual field,and relative size. The ground plane geometry can be



Filtering Mapping

AR view

Registration Data

Camera Image


X

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

Virtual Cue Geometry

Fig. 10: Pipeline for creating external virtual cues. An external geometry is added to the compositing step in order tocreate additional depth cues. The camera image is only needed for video see-through displays or when extracting otherimage features.

Fig. 11: Examples for using external virtual cues. Left)Adding a set of parallel lines to improve the depth percep-tion (Image courtesy of the U. S. Naval Research Laboratory,Livingston et al. [35] c©[2009] IEEE)). Right) Virtual ShadowPlanes for visualizing absolute depths (Wither et al. [63]c©[2005] IEEE).

extended by ties that show the connection betweenthe virtual object of interest and the ground plane.This feature is in particular interesting for floatingor subsurface objects, since it shows the connectionbetween these objects and the ground. Wither et al.introduced a similar concept with the Shadow Planes[63]. The shadow planes consist of two orthogonalplanes with depth measurements that are used toproject shadows of virtual objects onto it (Figure11, Right). The shadows in combination with a vir-tual distance scale on the planes was introduced tosupport the user in judging distances. Nevertheless,first study results showed no significant improvementusing this technique.

6.2.2 Appearance Cues

Less obstructive, but also less direct are methods thatencode the distance into the visual appearance. Thesemethods form the second group of virtual cues. InFigure 12, we show an instance of the AR visualiza-tion pipeline that reflects this mapping. The pipelineadds the distance from the user to the virtual objectto the mapping process and includes it into the visualappearance of the object. Thereby, the registration datahelps to compute the distance between virtual object

and user. Visual characteristics that are used to encodedistance are transparency, color, frequency of stipplesor density of virtual edges.

This kind of mapping was discussed by Livingstonet al. [112]. In their work, the authors suggested tochange opacity and intensity of building renderingsbased on their distance. They compared this visualmapping to constant opacity and constant intensityand found a significant effect of using decreasingopacity on depth estimation (Figure 13. Left). Urataniet al. discussed how to map monocular depth cues tothe appearance by using the distance of labels suchas [62]:

• Depth of field by blurring the frame of the labeldepending on the distance.

• Relative size by changing the size of the label’sframe.

• Aerial perspective by changing the saturation ofthe label as a function of distance.

• Texture gradient by including a texture patterninto the label.

In their final implementation they encoded the ab-solute distance into a color pattern (Figure 13, Right).More recently, Livingston et al. used a set of mappingtechniques to encode depth of virtual targets andcompared them to each other [66]. Mappings that theyused to encode the distance comprise:

• Stipples around the target, whereby the fre-quency increases with the distance.

• Opacity of the target that decreases with thedistance.

• Synthetic edges around the target, whereby thedistance is encoded in the spatial frequency ofthe edge pattern.

• Tunnel metaphor that uses squares around thetarget, whereby the number of squares dependson the number of occluding layers to the user(Figure 13, Right).

In a user study with professional users from the



Filtering

AR view

Camera Image


X

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

Registration Data

Mapping

Mapping

Fig. 12: Pipeline for mapping distance to appearance. The camera image is only needed for video see-through displays orwhen extracting other image features.

Proc. of Intl. Symposium on Mixed and Augmented Reality (Tokyo, Japan) 7–10 Oct 2003

Figure 2. User’s view of the stimuli. Left: “wire” drawing style. Center: “fill” drawing style. Right:“wire+fill” drawing style. The target (smallest, most central box) is between (position “middle”)obstructions 2 and 3 in all three pictures. These pictures were acquired by placing a camera to theeyepiece of the HMD, which accounts for the poor image quality. The vignetting and distortion aredue to the camera lens and the fact that it does not quite fit in the exit pupil of the HMD’s optics.

Drawing Style (“wire”, “fill”, “wire+fill”): Although thesame geometry was visible in each stimulus (except forwhich target was shown), the representation of that geom-etry was changed to determine what effect it had on depthperception. We used three drawing styles (Figure 2). Inthe first, all objects are drawn as wireframe outlines. Inthe second, the first (physically visible) object is drawn as awireframe outline, and all other objects are drawnwith solidfill (with no wireframe outline). In the third style, the firstobject is in wireframe, and all other layers are drawn withsolid fill with a white wireframe outline. Backface cullingwas on for all drawing styles, so that the user saw only twofaces of any occluded building.

Opacity (constant, decreasing): We designed two sets ofvalues for the ! channel based on the number of occludingobjects. In the “constant” style, the first layer (visible withregistered wireframe outline) is completely opaque, and allother layers have the same opacity (! 0 5). In the “de-creasing” style, opacity changes for each layer. The first(physically visible, wireframe) layer is completely opaque.The successive layers are not opaque; the ! values were 0 6,0 5, and 0 4 for the successively more distant layers.

Intensity (constant, decreasing): We used two sets of in-tensity modulation values. The modulation value was ap-plied to the object color (in each color channel, but not in theopacity or ! channel) for the object in the layer for which itwas specified. In the “constant” style, the first layer (visiblewith registered wireframe outline) has full intensity (modu-lator=1.0) and all other layers have intensity modulator=0.5.In the “decreasing” style, the first layer has its full native in-tensity, but successive layers are modulated as a function ofoccluding layers: 0.75 for the first, 0.50 for the second, and0.25 for the third (final) layer.

Target Position (close, middle, far): As shown in theoverhead map view (Figure 3), there were three possiblelocations for the target.

Figure 3. The experimental design (not toscale) shows the user position at the left.Obstruction 1 denotes the visible surfacesof the physically visible building. The dis-tance from the user to obstruction 1 is ap-proximately 60 meters. The distance fromthe user to target location 3 is approximately500 meters, with the obstructions and targetlocations roughly equally spaced.

Ground Plane (on, off): From the literature and every-day experience, we know that the perspective effects of theground plane rising to meet the horizon and apparent objectsize are a strong depth cues. In order to test the representa-tions as an aide to depth ordering, we removed the groundplane constraint in half of the trials. The building sizes werechosen to have the same apparent size from the users’ loca-tion for all trials. When the ground plane constraint wasnot present in the stimulus, the silhouette of each target wasfixed for a given pose of the user. In other words, targetstwo and three were not only scaled (to yield the same ap-parent size) but also positioned vertically such that all threetargets would occupy the same pixels on the 2D screen forthe same viewing position and orientation. No variation inposition with respect to the two horizontal dimensions wasnecessary when changing from using the ground plane tonot using it. The obstructions were always presented withthe same ground plane. We informed the users for which

4

Fig. 13: Mapping from distance to appearance. Left) Us-ing intensity and opacity for encoding depth (image cour-tesy of the U. S. Naval Research Laboratory, Livingstonet al. [112] c©[2003] IEEE). Right) Using color to encodedistance (Uratani et al. [62], c©[2005] IEEE).

military, Livingston et al. compared the cues by askingthe participants to map the virtual targets to fivedepth zones. The results of the study showed theTunnel metaphor to be the most helpful cue.

Another more recent work, investigated how ran-dom dot patterns can be used to generate virtualdepth cues to support depth understanding in astereoscopic AR setting [99].

While the mapping metaphors can provide absoluteor relative depth measurements, they are usually notso well designed for maintaining visual coherence. Inthe next paragraph, we will show virtual cues aimsupport both.

A special type of virtual depth cues are cutaways.Cutaways are visualization techniques that focus onsupporting depth perception for occluded data. Forthis purpose, a part of the occluding object is cutout and reveals the hidden content. Cutaways areoften considered as being part of the group of Fo-cus&Context techniques, since they allow one to in-spect data in the cutaway area more in detail. Butactually they can do more than filtering. They are alsoable to provide virtual depths cues, such as a boxaround the hidden object that shows measurementsor perspective cues given by the shape of the cutoutgeometry. In contrast to using external geometriesor mapping distance to appearance, the creation of

cutaways requires extensive information about thephysical world. Similar to the ghosting techniques,cutaways have their origin in illustrations and techni-cal drawings, where the artist wants to reveal hiddenparts of an object to the observer.

The input that is required to create a convincingcutaway in AR comprises a cutout geometry as wellas a model of the occluding object. Since the occludingobject is in this case the physical world, we need con-textual data about the physical world (Figure 14). Thisdata could be a rough surface model or a phantomgeometry. By combining the cutout geometry and thephantom model, we can compute the correct cutoutby aligning the cutout to the surface of the physicalworld object.

In their research from 2002, Furmanski et al. dis-cussed general guidelines for designing X-Ray ARsystems [59]. Among different suggestions for visualaids (ground planes grids, distance marker and tem-poral distance markers), they showed how to rendervirtual cutaways on a wall to reveal hidden informa-tion. In a user study, they compared the visualizationof a target inside a wall with and without cutaways.Contrary to the expectations, the study showed thatthe virtual cutaways do only help to understand thelocation of the virtual target for a dynamic videosequence, where the target was partially occludedby the frame of the cutaway box. But the authorsstated that the findings from their study can be in-fluenced by technical limitations of the AR system.This was confirmed by the participants reporting thattheir perception was influenced by the jitter from theregistration.

Later on, Kalkofen used cutaways to visualize theinterior of a miniature car. They used a phantomrepresentation of the occluding object (the car) tocompute the bending of the contour of the cut-outarea. Based on this information, they were able topreserve the shape of the occluding object [47]. Fur-



Filtering Mapping

AR view

Registration Data

Camera Image


X

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

Cutout geometry

Compositing

Contextual Data

Fig. 14: Creating additional virtual cues with cutaway geometries. The camera image is only needed for video see-throughdisplays or when extracting other image features.

Fig. 15: Cutaways as virtual cues in AR. A virtual exca-vation with a depth scale is used to visualize subsurfaceinfrastructure (Zollmann et al. [113]).

ther, the work of Kalkofen showed that the cutoutis not enough to transfer the depth of a detachedhidden object. In this case, the visualization techniqueshould provide additional hidden aids or geometries.Kalkofen, addressed this by rendering the cutout vol-ume to add depth cues.

Mendez et al. showed how to include such addi-tional visual hints in a cutaway visualization [114].They rendered a virtual excavation with depth mea-surements to visualize subsurface infrastructure inan urban civil engineering scenario. The virtual boxallows the user to estimate the depth of the hiddenobjects. Furthermore, occlusions between the virtualpipes and the textured box allow to support the depthperception, since it shows the exact spatial relation-ship between the cutout geometry and an object ofinterest.

This method was automatized by Zollmann et al.with a dynamically configurable transcoding methodthat allows for generating cutaway geometries auto-matically from GIS data [113] (Figure 15).

6.3 Filtering PipelineThe main goal of information filtering is to reducethe complexity by decreasing the amount of displayedinformation based on a defined logic. In AR, location,user objectives and user-defined focus areas were

used to control the filtering (Figure 16). One of theearly research works that investigated filtering in ARis the work of Julier et al. [72]. They proposed a sys-tem for reducing information clutter in a mobile ARsystem by calculating a focus and nimbus area basedon the user’s location and objectives. Based on thisinformation, they calculated the importance of virtualbuildings and used it to decide whether a virtualbuilding should be culled or not. A fading functionprovides smooth transitions between filtering levels toavoid that small changes in user’s positions extremelychange the displayed content. Later, Livingston etal. used a similar filtering approach based on focusand nimbus areas of objects of interests for removingclutter in military AR operations [66].

Focus&Context techniques in AR allow to filtervirtual information based on an interactively definedspatial logic. For instance, Looser et al. introducedan interactive magic lens for defining a focus area[115]. Users of their system can control the lens witha physical marker. Based on this selection, only thearea inside the lens displays virtual data. This allowsthe user to inspect the virtual data while avoiding acluttered context area. Additionally, the filtering crite-ria of their magic lens tool can be configured duringrun-time. Other interactive Focus&Context tool arethe interactive X-Ray tunnel and the room-selectortool from Bane and Hollerer [74]. These tools allowdefining a focus area that display virtual data, suchas heat distribution of a building (Figure 17, Left).Kalkofen et al. used Focus&Context filters in a scenegraph to allow users to explore occluded informationin an X-Ray view (Figure 17, Right) [29].

6.4 Data Transformation PipelineThe drawback of filtering techniques is that theyeliminate information to avoid information clutter.In contrast, transformation techniques do not com-pletely remove non-relevant information, but trans-



Filtering Mapping

AR view

Camera Image

Registration Data


X

AR specific steps

AR specific input

Normal Vis-Pipeline

Legend

Compositing

Interactive Input

Fig. 16: Information filtering pipeline. The camera image is only needed for video see-through displays or when extractingother image features.

Fig. 17: Filtering pipeline examples using Focus&Contexttechniques for information filtering in AR. Left) InteractiveX-Ray tunnel (Bane et al. [74], c©[2004] IEEE). Right) Fo-cus&Context tools allow to explore occluded informationin an X-Ray view (Kalkofen et al. [29] c©[2007] IEEE).

form it. Transformation techniques were developedin the context of information visualization, but werealso already applied for AR visualizations. In AR,either the physical world information or the virtualinformation is transformed to create a more cleanvisualization. For instance, inspired by illustrativetechniques Kalkofen et al. created explosion views toremove occluding areas from a hidden object of inter-est [77]. Their technique translates occluding parts to anew position to reveal occluded information. Recently,Tatzgern et al. extended these techniques by usingcompact explosion views to avoid that transformedcontent infers with the environment [78].

Other examples for transformation techniques aredistortions. Distortion methods have been used ininformation visualization and interface design [116].Those methods aim to reduce the space used upfor the visualization of non-relevant information bydistorting it allowing to keep non-relevant informa-tion available for a fast overview. Sandor et al. useda distortion of occluding physical world objects toreveal occluded objects [117].

While the previous transformation methods ma-nipulate the appearance of the physical world, ap-proaches that focus on view management for labelplacement often move the actual labels in a way thatall information is readable. While the techniques ofBell et al. and Azuma et al. manipulate the label

placement based on the label characteristics or the re-lationship between labels [73], [118], the techniques byRosten et al. and Grasset et al. apply transformation tovirtual annotations in order to rearrange them basedon an analysis of the current environment [119], [81].

Another option is to transform the appearance. Forinstance, ElSayed et al. [95] used colors and dimen-sions of 3D models to represent certain intensity ofan ingredient in a food product.

7 CONCLUSION

In this paper, we analysed and classified related workin the field of visualization for AR. We describedhow the visualization techniques support differentpurpose in AR applications. With our classification,we were able to group similar techniques and toidentify common design streams and pipelines. Wealso discussed how AR visualization techniques spanthrough their primary purpose, the visibility of data,the depth cues used, filtering or abstraction of dataand lastly the compositing methods.

When investigating the different visualizationpipelines for AR, it became clear that contextualinformation about the physical environment is of-ten required. This is in particular a challenge foroutdoor usage or large scale environments, such assport events. A lot of existing visualization methodsin AR focus on indoor usage or small work spaces,which usually have predictable lighting and rely ona familiar environment of the physical world. Severalmethods even assume that a complete model of thephysical world is available, which is a difficult featfor outdoor environments due to the sheer size anddynamic variables outdoors.

We also found that visual coherence seems to be themost commonly addressed aim of the investigated ARvisualization techniques. Depth and occlusion cuesare vital components to achieve visual coherence inAR applications. In particular X-Ray visualizationshighly benefit from the integration of depth cues.Existing methods often focus either on extracting


occlusion cues from a camera image or are extractingsuch cues from a 3D model, but therefore relying onaccurately modelled environment. Combining both,image-based cues and model-based cues seems to bea promising area that is still under-explored.

While support for depth perception, reducing infor-mation clutter and exploration are similarly popularpurposes for AR visualization techniques, attentiondirection has only been explored in a smaller numberof research works and has potential for further in-vestigations. In particular here, aspects of how to usevirtual and physical perceptual cues could be used forguiding attention could be interesting to investigatefurther.

Finally, we should state again that we focus ongeneral visualization issues in AR. As pointed outearlier, specific AR displays might require additionalsteps specific to this display to achieve some of thegoals outlined in this work. In particular optical see-through displays pose their own challenges as they donot give full control over the environment as assumedin this work and possible with video see-through AR(e.g. rendering correct occlusions is challenging withcurrent hardware as it is only possible to add light[7]).

8 FUTURE WORKIn the context of creating a taxonomy and reviewingrelated work in AR, there are still unexplored fields inAR that are not included in this paper. Applicationswith huge databases might need a more complex andautomated way to visualize virtual cues to supportdepth perception. Complex multidimensional data inAR is also a field which is still not properly investi-gated. What are the visualization methods to visualize4D representations where users could understand therelationship between the multiple datasets? Also theInternet of Things (IoT) is definitely one of the mostpromising future application areas for AR that couldbenefit from research in visualisation techniques inAR. IoT devices are embedded in our environmentand besides a mobile app they often do not showthe captured information. AR and in particular so-phisticated visualisation techniques could provide anintuitive interface to the large amount of rich dataused by the IoT. This trend is among other discussedby Norouzi et al. [120] but they envision an intelligentvirtual agent representing the data while visualisationtechniques as presented in this paper could make theraw data more accessible.

As for future AR research itself, we hope our ARvisualization pipelines and taxonomy contribute toany future AR applications. We highlighted on howvisualization techniques currently are often limitedto indoor usage. Considerably, more work needs tobe done in order to achieve a robust AR systemthat could function well indoors and outdoors whileproviding visual coherent visualizations.

ACKNOWLEDGMENTSThis work is supported by a MBIE Smart Ideas projectgrant and a L’Oreal/UNESCO FWIS fellowship.

REFERENCES[1] R. Haber and D. McNaab, “Visualization idioms: a conceptual

model for scientific visualization systems,” IEEE ComputerSociety Press, 1990.

[2] K. Kim, M. Billinghurst, G. Bruder, H. Been-Lirn Duh, andG. F. Welch, “Revisiting trends in augmented reality research:A review of the 2nd decade of ismar (20082017),” IEEETransactions on Visualization and Computer Graphics, vol. PP,pp. 1–1, 09 2018.

[3] D. Kalkofen, C. Sandor, S. White, and D. Schmalstieg,Visualization Techniques for Augmented Reality. New York,NY: Springer New York, 2011, pp. 65–98. [Online]. Available:https://doi.org/10.1007/978-1-4614-0064-6 3

[4] R. T. Azuma, “A survey of augmented reality,”Presence: Teleoperators and Virtual Environments,vol. 6, no. 4, pp. 355–385, 1997. [Online].Available: http://nzdis.otago.ac.nz/projects/projects/berlin/repository/revisions/22/raw/trunk/Master’sDocs/Papers/ASurveyofAugmentedReality.pdf

[5] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, andB. MacIntyre, “Recent advances in augmented reality,” IEEEcomputer graphics and applications, vol. 21, no. 6, pp. 34–47,2001.

[6] J. L. Gabbard, J. E. Swan, J. Zedlitz, and W. W. Winchester,“More than meets the eye: An engineering study to empiri-cally examine the blending of real and virtual color spaces,”in 2010 IEEE Virtual Reality Conference (VR). IEEE, 2010, pp.79–86.

[7] T. Langlotz, M. Cook, and H. Regenbrecht, “Real-time radio-metric compensation for optical see-through head-mounteddisplays,” IEEE transactions on visualization and computergraphics, vol. 22, no. 11, pp. 2385–2394, 2016.

[8] Y. Itoh, T. Hamasaki, and M. Sugimoto, “Occlusion leakcompensation for optical see-through displays using a single-layer transmissive spatial light modulator,” IEEE transactionson visualization and computer graphics, vol. 23, no. 11, pp. 2463–2473, 2017.

[9] J. Collins, H. Regenbrecht, and T. Langlotz, “Visual coherencein mixed reality: A systematic enquiry,” Presence: Teleoperatorsand Virtual Environments, vol. 26, no. 1, pp. 16–41, 2017.[Online]. Available: https://doi.org/10.1162/PRES a 00284

[10] E. Kruijff, J. E. Swan, and S. Feiner, “Perceptual issues in aug-mented reality revisited,” in IEEE International Symposium onMixed and Augmented Reality (ISMAR 2010). IEEE, Oct. 2010,pp. 3–12. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=5643530’escapeXml=’false’/〉

[11] S. Zollmann, R. Grasset, G. Reitmayr, and T. Langlotz,“Image-based x-ray visualization techniques for spatial un-derstanding in outdoor augmented reality,” in Proceedings ofthe 26th Australian Computer-Human Interaction Conference onDesigning Futures: the Future of Design. ACM, 2014, pp. 194–203.

[12] E. A. Bier, M. C. Stone, K. Fishkin, W. Buxton, andT. Baudel, “A taxonomy of see-through tools,” in Proceedingsof the SIGCHI Conference on Human Factors in ComputingSystems, ser. CHI ’94. New York, NY, USA: ACM, 1994, pp.358–364. [Online]. Available: http://doi.acm.org/10.1145/191666.191786

[13] D. A. Bowman, C. North, J. Chen, N. F. Polys, P. S. Pyla, andU. Yilmaz, “Information-Rich Virtual Environments: Theory,Tools, and Research Agenda,” Proceedings of the ACM sympo-sium on Virtual reality software and technology, 2003.

[14] D. A. Bowman and L. F. Hodges, “Formalizing the design,evaluation, and application of interaction techniques for im-mersive virtual environments,” Journal of Visual Languages andComputing, 1999.

[15] N. Elmqvist and P. Tsigas, “A taxonomy of3D occlusion management for visualization.” IEEEtransactions on visualization and computer graphics, vol. 14,no. 5, pp. 1095–109, Jan. 2008. [Online]. Available:http://dl.acm.org/citation.cfm?id=1446227.1446259

https://doi.org/10.1007/978-1-4614-0064-6_3

http://nzdis.otago.ac.nz/projects/projects/berlin/repository/revisions/22/raw/trunk/Master's Docs/Papers/A Survey of Augmented Reality.pdf



https://doi.org/10.1162/PRES_a_00284

">http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5643530' escapeXml='false'/>

">http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5643530' escapeXml='false'/>

http://doi.acm.org/10.1145/191666.191786

http://doi.acm.org/10.1145/191666.191786

http://dl.acm.org/citation.cfm?id=1446227.1446259


[16] F. Zhou, H. B. Duh, and M. Billinghurst, “Trends in aug-mented reality tracking, interaction and display: A reviewof ten years of ismar,” in 2008 7th IEEE/ACM InternationalSymposium on Mixed and Augmented Reality, Sep. 2008, pp.193–202.

[17] M. Billinghurst, A. Clark, G. Lee, and Others, “A surveyof augmented reality,” Foundations and Trends R©in Human–Computer Interaction, vol. 8, no. 2-3, pp. 73–272, 2015.

[18] K. Kim, M. Billinghurst, G. Bruder, H. B. Duh, and G. F.Welch, “Revisiting trends in augmented reality research: Areview of the 2nd decade of ismar (20082017),” IEEE Trans-actions on Visualization and Computer Graphics, vol. 24, no. 11,pp. 2947–2962, Nov 2018.

[19] D. Yu, J. S. Jin, S. Luo, W. Lai, and Q. Huang, “AUseful Visualization Technique: A Literature Review forAugmented Reality and its Application, limitation & futuredirection,” in Visual Information Communication. Boston,MA: Springer US, 2009, pp. 311–337. [Online]. Available:http://link.springer.com/10.1007/978-1-4419-0312-9 21

[20] W. Willett, Y. Jansen, and P. Dragicevic, “Embedded data rep-resentations,” IEEE Transactions on Visualization and ComputerGraphics, vol. 23, no. 1, pp. 461–470, Jan 2017.

[21] E. K. K. Yasojima, B. S. Meiguins, and A. S. Meiguins,“Collaborative augmented reality application for informationvisualization support,” in 2011 15th International Conference onInformation Visualisation, July 2011, pp. 170–175.

[22] O. Bimber and R. Raskar, Spatial augmented reality: mergingreal and virtual worlds. AK Peters/CRC Press, 2005.

[23] D. Wagner, T. Langlotz, and D. Schmalstieg, “Robust andunobtrusive marker tracking on mobile phones,” in 2008 7thIEEE/ACM International Symposium on Mixed and AugmentedReality. IEEE, 2008, pp. 121–124.

[24] G. Klein and D. Murray, “Parallel tracking and mapping forsmall ar workspaces,” in Proceedings of the 2007 6th IEEE andACM International Symposium on Mixed and Augmented Reality.IEEE Computer Society, 2007, pp. 1–10.

[25] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, andD. Schmalstieg, “Pose tracking from natural features on mo-bile phones,” in Proceedings of the 7th IEEE/ACM internationalsymposium on mixed and augmented reality. IEEE ComputerSociety, 2008, pp. 125–134.

[26] D. Wagner, A. Mulloni, T. Langlotz, and D. Schmalstieg,“Real-time panoramic mapping and tracking on mobilephones,” in 2010 IEEE virtual reality conference (VR). IEEE,2010, pp. 211–218.

[27] J. Ventura, C. Arth, G. Reitmayr, and D. Schmalstieg, “Globallocalization from monocular slam on a mobile phone,” IEEEtransactions on visualization and computer graphics, vol. 20,no. 4, pp. 531–539, 2014.

[28] J. Grubert, T. Langlotz, S. Zollmann, and H. Regenbrecht,“Towards pervasive augmented reality: Context-awarenessin augmented reality,” IEEE transactions on visualization andcomputer graphics, vol. 23, no. 6, pp. 1706–1724, 2016.

[29] D. Kalkofen, E. Mendez, and D. Schmalstieg, “InteractiveFocus and Context Visualization for Augmented Reality,” inIEEE International Symposium on Mixed and Augmented Reality(ISMAR 2007), Nov. 2007, pp. 191–200.

[30] D. E. Breen, R. T. Whitaker, E. Rose, and M. Tuceryan,“Interactive Occlusion and Automatic Object Placement forAugmented Reality,” Computer Graphics Forum, vol. 15,no. 3, pp. 11–22, Aug. 1996. [Online]. Available: http://doi.wiley.com/10.1111/1467-8659.1530011

[31] S. Zollmann, D. Kalkofen, E. Mendez, and G. Reitmayr,“Image-based ghostings for single layer occlusions inaugmented reality,” in IEEE International Symposium onMixed and Augmented Reality (ISMAR 2010). IEEE, Oct.2010, pp. 19–26. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5643546

[32] B. Avery, C. Sandor, and B. H. Thomas, “Improving spatialperception for augmented reality x-ray vision,” in 2009 IEEEVirtual Reality Conference, March 2009, pp. 79–82.

[33] T. Fukiage, T. Oishi, and K. Ikeuchi, “Visibility-based blend-ing for real-time applications,” in ISMAR 2014 - IEEE Inter-national Symposium on Mixed and Augmented Reality - Scienceand Technology 2014, Proceedings, 2014.

[34] J. E. Cutting, “How the eye measures reality and virtual

reality,” Behavior Research Methods, Instruments, & Computers,vol. 29, no. 1, pp. 27–36, 1997.

[35] M. Livingston and Z. Ai, “Indoor vs. outdoor depthperception for mobile augmented reality,” in IEEEVirtual Reality Conference (VR 2009), 2009, pp. 55–62.[Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=4810999

[36] R. Rosenholtz, Y. Li, J. Mansfield, and Z. Jin, “Featurecongestion: a measure of display clutter,” Proceedings of theSIGCHI conference on Human factors in computing systems,pp. 761–770, 2005. [Online]. Available: http://dl.acm.org/citation.cfm?id=1055078

[37] S. Nolle and G. Klinker, “Augmented reality as a comparisontool in automotive industry,” in IEEE/ACM InternationalSymposium on Mixed and Augmented Reality. Ieee, 2006,pp. 249–250. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4079291

[38] K. Copic Pucihar, P. Coulton, and J. Alexander, “The use ofsurrounding visual context in handheld AR,” in Proceedings ofthe 32nd annual ACM conference on Human factors in computingsystems - CHI ’14, 2014.

[39] V. Lepetit and M. . Berger, “A semi-automatic method forresolving occlusion in augmented reality,” in Proceedings IEEEConference on Computer Vision and Pattern Recognition. CVPR2000 (Cat. No.PR00662), vol. 2, June 2000, pp. 225–230 vol.2.

[40] S. Zokai, J. Esteve, Y. Genc, and N. Navab, “Multiviewparaperspective projection model for diminished reality,” inThe Second IEEE and ACM International Symposium on Mixedand Augmented Reality, 2003. Proceedings., Oct 2003, pp. 217–226.

[41] J. Fischer, D. Bartz, and W. Strasser, “Occlusion handlingfor medical augmented reality using a volumetric phantommodel,” in Proceedings of the ACM Symposium on VirtualReality Software and Technology, ser. VRST ’04. New York,NY, USA: ACM, 2004, pp. 174–177. [Online]. Available:http://doi.acm.org/10.1145/1077534.1077570

[42] J. D. Mulder, “Realistic occlusion effects in mirror-based co-located augmented reality systems,” in IEEE Proceedings. VR2005. Virtual Reality, 2005., March 2005, pp. 203–208.

[43] P. Fortin and P. Hebert, “Handling occlusions in real-timeaugmented reality : Dealing with movable real and virtualobjects,” in The 3rd Canadian Conference on Computer and RobotVision (CRV’06), June 2006, pp. 54–54.

[44] M. Fiala, “Dark matter method for correct augmented realityocclusion relationships,” in 2006 IEEE International Workshopon Haptic Audio Visual Environments and their Applications(HAVE 2006), 2006, pp. 90–93.

[45] C. Bichlmeier, F. Wimmer, H. Sandro Michael, and N. Nassir,“Contextual Anatomic Mimesis: Hybrid In-Situ VisualizationMethod for Improving Multi-Sensory Depth Perception inMedical Augmented Reality,” in IEEE International Symposiumon Mixed and Augmented Reality (ISMAR 2007), 2007, pp. 129–138.

[46] J. Zhu and Z. Pan, “Occlusion registration in video-basedaugmented reality,” in Proceedings of The 7th ACM SIGGRAPHInternational Conference on Virtual-Reality Continuum and ItsApplications in Industry, 2008, pp. 1–6.

[47] D. Kalkofen, E. Mendez, and D. Schmalstieg, “Interactivefocus and context visualization for augmented reality,” in2007 6th IEEE and ACM International Symposium on Mixed andAugmented Reality. IEEE, 2007, pp. 191–201.

[48] F. I. Cosco, C. Garre, F. Bruno, M. Muzzupappa, and M. A.Otaduy, “Augmented Touch Without Visual Obtrusion,” inProceedings of the 2009 8th IEEE International Symposium onMixed and Augmented Reality, ser. ISMAR ’09. Washington,DC, USA: IEEE Computer Society, 2009, pp. 99–102. [Online].Available: http://dx.doi.org/10.1109/ISMAR.2009.5336492

[49] J. Chen, X. Granier, N. Lin, and Q. Peng, “On-linevisualization of underground structures using contextfeatures,” in Proceedings of the 17th ACM Symposium onVirtual Reality Software and Technology, ser. VRST ’10. NewYork, NY, USA: ACM, 2010, pp. 167–170. [Online]. Available:http://doi.acm.org/10.1145/1889863.1889898

[50] C. Sandor, A. Cunningham, A. Dey, and V. Mattila,“An Augmented Reality X-Ray system based on visualsaliency,” in IEEE International Symposium on Mixed and

http://link.springer.com/10.1007/978-1-4419-0312-9_21

http://doi.wiley.com/10.1111/1467-8659.1530011

http://doi.wiley.com/10.1111/1467-8659.1530011

http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5643546


http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4810999


http://dl.acm.org/citation.cfm?id=1055078




http://doi.acm.org/10.1145/1077534.1077570

http://dx.doi.org/10.1109/ISMAR.2009.5336492

http://doi.acm.org/10.1145/1889863.1889898


Augmented Reality (ISMAR 2010). IEEE, 2010, pp. 27–36.[Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=5643547

[51] O. Pauly, B. Diotte, S. Habert, S. Weidert, E. Euler, P. Fallavol-lita, and N. Navab, “Relevance-based visualization to im-prove surgeon perception,” in Information Processing inComputer-Assisted Interventions, D. Stoyanov, D. L. Collins,I. Sakuma, P. Abolmaesumi, and P. Jannin, Eds. Cham:Springer International Publishing, 2014, pp. 178–185.

[52] D. Kalkofen, E. Veas, S. Zollmann, M. Steinberger, andD. Schmalstieg, “Adaptive ghosted views for augmentedreality,” in 2013 IEEE International Symposium on Mixed andAugmented Reality (ISMAR), Oct 2013, pp. 1–9.

[53] N. Kawai, M. Yamasaki, T. Sato, and N. Yokoya, “DiminishedReality for AR Marker Hiding Based on Image Inpaintingwith Reflection of Luminance Changes,” ITE Transactions onMedia Technology and Applications, 2013.

[54] F. Cosco, C. Garre, F. Bruno, M. Muzzupappa, and M. A.Otaduy, “Visuo-haptic mixed reality with unobstructed tool-hand integration,” IEEE Transactions on Visualization and Com-puter Graphics, vol. 19, no. 1, pp. 159–172, Jan 2013.

[55] K. Sugimoto, H. Fujii, A. Yamashita, and H. Asama, “Half-diminished reality image using three rgb-d sensors for remotecontrol robots,” 2014 IEEE International Symposium on Safety,Security, and Rescue Robotics (2014), pp. 1–6, 2014.

[56] M. Roxas, T. Hori, T. Fukiage, Y. Okamoto, and T. Oishi,“Occlusion handling using semantic segmentation andvisibility-based rendering for mixed reality,” in Proceedingsof the 24th ACM Symposium on Virtual Reality Software andTechnology, ser. VRST ’18. New York, NY, USA: ACM,2018, pp. 20:1–20:8. [Online]. Available: http://doi.acm.org/10.1145/3281505.3281546

[57] G. Klein and D. W. Murray, “Simulating Low-Cost Camerasfor Augmented Reality Compositing,” Visualization and Com-puter Graphics, IEEE Transactions on, vol. 16, no. 3, pp. 369–380,2010.

[58] M. Bajura, H. Fuchs, and R. Ohbuchi, “Merging virtualobjects with the real world: Seeing ultrasound imagery withinthe patient,” in Proceedings of the 19th Annual Conference onComputer Graphics and Interactive Techniques, ser. SIGGRAPH’92. New York, NY, USA: ACM, 1992, pp. 203–210. [Online].Available: http://doi.acm.org/10.1145/133994.134061

[59] C. Furmanski, R. Azuma, and M. Daily, “Augmented-Reality Visualizations Guided by Cognition: PerceptualHeuristics for Combining Visible and Obscured Information,”IEEE International Symposium on Mixed and AugmentedReality (ISMAR 2002), 2002. [Online]. Available: http://portal.acm.org/citation.cfm?id=850976.854989

[60] G. R. King, W. Piekarski, and B. H. Thomas, “ARVino -Outdoor Augmented Reality Visualisation of Viticulture GISData,” in IEEE International Symposium on Mixed and Aug-mented Reality (ISMAR 2005), 2005, pp. 52–55.

[61] T. Tsuda, H. Yamamoto, Y. Kameda, and Y. Ohta,“Visualization methods for outdoor see-through vision,”in Proceedings of the 2005 International Conference onAugmented Tele-existence, ser. ICAT ’05. New York,NY, USA: ACM, 2005, pp. 62–69. [Online]. Available:http://doi.acm.org/10.1145/1152399.1152412

[62] K. Uratani and T. Machida, “A study of depth visualizationtechniques for virtual annotations in augmented reality,”in IEEE Virtual Reality Conference (VR 2005), 2005.[Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=1492802

[63] J. Wither and T. Hollerer, “Pictorial depth cues for outdooraugmented reality,” in Ninth IEEE International Symposium onWearable Computers (ISWC’05), Oct 2005, pp. 92–99.

[64] N. Navab, M. Feuerstein, and C. Bichlmeier, “Laparoscopicvirtual mirror new interaction paradigm for monitor basedaugmented reality,” in 2007 IEEE Virtual Reality Conference,March 2007, pp. 43–50.

[65] G. Schall, E. Mendez, and D. Schmalstieg, “Virtual redlin-ing for civil engineering in real environments,” in 2008 7thIEEE/ACM International Symposium on Mixed and AugmentedReality, Sep. 2008, pp. 95–98.

[66] M. A. Livingston, Z. Ai, K. Karsch, and G. O. Gibson, “Userinterface design for military ar applications,” Virtual Reality,

vol. 15, no. 2, pp. 175–184, Jun 2011. [Online]. Available:https://doi.org/10.1007/s10055-010-0179-1

[67] M. T. Eren, M. Cansoy, and S. Balcisoy, “Multi-view aug-mented reality for underground exploration,” in 2013 IEEEVirtual Reality (VR), March 2013, pp. 117–118.

[68] M. Kyto, A. Makinen, J. Hakkinen, and P. Oittinen, “Improv-ing relative depth judgments in augmented reality with aux-iliary augmentations,” ACM Transactions on Applied Perception(TAP), vol. 10, no. 1, p. 6, 2013.

[69] S. Zollmann, C. Hoppe, T. Langlotz, and G. Reitmayr, “Flyar:Augmented reality supported micro aerial vehicle naviga-tion,” IEEE Transactions on Visualization and Computer Graphics,vol. 20, no. 4, pp. 560–568, April 2014.

[70] P. Skinner, J. Ventura, and S. Zollmann, “Indirect augmentedreality browser for gis data,” in 2018 IEEE InternationalSymposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2018, pp. 145–150.

[71] S. Mann, “Mediated reality,” M.I.T. M.L. Technical Report 260,Cambridge, Massachusetts, Tech. Rep., 1994.

[72] S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner,T. Hollerer, and S. Sestito, “Information filtering formobile augmented reality,” in Proceedings IEEE and ACMInternational Symposium on Augmented Reality ISAR 2000.IEEE COMPUTER SOC, 2000, pp. 3–11. [Online]. Available:http://discovery.ucl.ac.uk/135575/

[73] B. Bell, S. Feiner, and T. Hollerer, “View managementfor virtual and augmented reality,” in Proceedings of the14th annual ACM symposium on User interface softwareand technology - UIST ’01. New York, New York, USA:ACM Press, Nov. 2001, p. 101. [Online]. Available: http://dl.acm.org/citation.cfm?id=502348.502363

[74] R. Bane and T. Hollerer, “Interactive Tools for Virtual X-RayVision in Mobile Augmented Reality,” in IEEE InternationalSymposium on Mixed and Augmented Reality (ISMAR 2004).IEEE, 2004, pp. 231–239. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=1383060

[75] K. Tanaka, Y. Kishino, M. Miyamae, T. Terada, and S. Nishio,“An information layout method for an optical see-throughhead mounted display focusing on the viewability,” inProceedings of the 7th IEEE/ACM International Symposium onMixed and Augmented Reality, ser. ISMAR ’08. Washington,DC, USA: IEEE Computer Society, 2008, pp. 139–142.[Online]. Available: https://doi.org/10.1109/ISMAR.2008.4637340

[76] C. Sandor, A. Cunningham, U. Eck, D. Urquhart, G. Jarvis,A. Dey, S. Barbier, M. Marner, and S. Rhee, “Egocentric Space-Distorting Visualizations for Rapid Environment Explorationin Mobile Mixed Reality,” in IEEE Virtual Reality Conference(VR 2010), 2010, pp. 47–50.

[77] D. Kalkofen, M. Tatzgern, and D. Schmalstieg, “ExplosionDiagrams in Augmented Reality,” IEEE Virtual Reality Con-ference (VR 2009), vol. 0, pp. 71–78, 2009.

[78] M. Tatzgern, D. Kalkofen, and D. Schmalstieg, “DynamicCompact Visualizations for Augmented Reality,” inIEEE Virtual Reality Conference (VR 2013), 2013.[Online]. Available: http://www.icg.tugraz.at/publications/dynamic-compact-visualizations-for-augmented-reality

[79] T. Kawamata, I. Kitahara, Y. Kameda, and Y. Ohta, “Poster:Lifted road map view on windshield display,” in 2013 IEEESymposium on 3D User Interfaces (3DUI), March 2013, pp. 139–140.

[80] S. Zollmann, D. Kalkofen, C. Hoppe, S. Kluckner,H. Bischof, and G. Reitmayr, “Interactive 4Doverview and detail visualization in augmentedreality,” in IEEE International Symposium on Mixedand Augmented Reality (ISMAR 2012), 2012. [Online].Available: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6402554&contentType=Conference+Publications&queryText=Interactive+4D+Overview+and+Detail+Visualization+in+Augmented+Reality

[81] R. Grasset, T. Langlotz, D. Kalkofen, M. Tatzgern,and D. Schmalstieg, “Image-Driven View Man-agement for Augmented Reality Browsers,” inIEEE International Symposium on Mixed and Aug-mented Reality (ISMAR 2012), 2012. [Online]. Avail-able: http://www.academia.edu/1966315/Image-DrivenView Management for Augmented Reality Browsers



http://doi.acm.org/10.1145/3281505.3281546

http://doi.acm.org/10.1145/3281505.3281546

http://doi.acm.org/10.1145/133994.134061

http://portal.acm.org/citation.cfm?id=850976.854989


http://doi.acm.org/10.1145/1152399.1152412



https://doi.org/10.1007/s10055-010-0179-1

http://discovery.ucl.ac.uk/135575/





https://doi.org/10.1109/ISMAR.2008.4637340


http://www.icg.tugraz.at/publications/dynamic-compact-visualizations-for-augmented-reality

http://www.icg.tugraz.at/publications/dynamic-compact-visualizations-for-augmented-reality

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6402554&contentType=Conference+Publications&queryText=Interactive+4D+Overview+and+Detail+Visualization+in+Augmented+Reality




http://www.academia.edu/1966315/Image-Driven_View_Management_for_Augmented_Reality_Browsers

http://www.academia.edu/1966315/Image-Driven_View_Management_for_Augmented_Reality_Browsers


[82] Y. Kameda, T. Takemasa, and Y. Ohta, “Outdoor see-throughvision utilizing surveillance cameras,” in Proceedings of the 3rdIEEE/ACM International Symposium on Mixed and AugmentedReality, ser. ISMAR ’04. Washington, DC, USA: IEEEComputer Society, 2004, pp. 151–160. [Online]. Available:https://doi.org/10.1109/ISMAR.2004.45

[83] S. White and S. Feiner, “Sitelens: Situated visualizationtechniques for urban site visits,” in Proceedings of theSIGCHI Conference on Human Factors in Computing Systems,ser. CHI ’09. New York, NY, USA: ACM, 2009, pp.1117–1120. [Online]. Available: http://doi.acm.org/10.1145/1518701.1518871

[84] T. Schinke, N. Henze, and S. Boll, “Visualization of off-screenobjects in mobile augmented reality,” in Proceedings of the12th International Conference on Human Computer Interactionwith Mobile Devices and Services, ser. MobileHCI ’10. NewYork, NY, USA: ACM, 2010, pp. 313–316. [Online]. Available:http://doi.acm.org/10.1145/1851600.1851655

[85] P. Jain, J. Manweiler, and R. Roy Choudhury, “Overlay:Practical mobile augmented reality,” in Proceedings of the13th Annual International Conference on Mobile Systems,Applications, and Services, ser. MobiSys ’15. New York,NY, USA: ACM, 2015, pp. 331–344. [Online]. Available:http://doi.acm.org/10.1145/2742647.2742666

[86] C. Barreiros, E. Veas, and V. Pammer-Schindler, “Pre-attentivefeatures in natural augmented reality visualizations,” in 2016IEEE International Symposium on Mixed and Augmented Reality(ISMAR-Adjunct), Sep. 2016, pp. 72–73.

[87] K. Karsch, M. Golparvar-Fard, and D. Forsyth,“Constructaide: Analyzing and visualizing construction sitesthrough photographs and building models,” ACM Trans.Graph., vol. 33, no. 6, pp. 176:1–176:11, Nov. 2014. [Online].Available: http://doi.acm.org/10.1145/2661229.2661256

[88] P. Perea, D. Morand, and L. Nigay, “Halo3d: A techniquefor visualizing off-screen points of interest in mobileaugmented reality,” in Proceedings of the 29th Conferenceon L’Interaction Homme-Machine, ser. IHM ’17. New York,NY, USA: ACM, 2018, pp. 43–51. [Online]. Available:http://doi.acm.org/10.1145/3132129.3132144

[89] U. Gruenefeld, A. E. Ali, S. Boll, and W. Heuten,“Beyond halo and wedge: Visualizing out-of-view objectson head-mounted virtual and augmented reality devices,”in Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services, ser.MobileHCI ’18. New York, NY, USA: ACM, 2018, pp.40:1–40:11. [Online]. Available: http://doi.acm.org/10.1145/3229434.3229438

[90] Z. Chen, Y. Su, Y. Wang, Q. Wang, H. Qu, and Y. Wu,“Marvist: Authoring glyph-based visualization in mobileaugmented reality,” IEEE Transactions on Visualization andComputer Graphics, pp. 1–1, 2019.

[91] S. C. R. D. C. F. Gutierrez, N. N. Htun and K. Verbert, “De-signing Augmented Reality Applications for Personal HealthDecision-Making,” in 52nd Hawaii International Conference onSystem Sciences, 2019.

[92] S. Mann and J. Fung, “Videoorbits on eye tap devices for de-liberately diminished reality or altering the visual perceptionof rigid planar patches of a real world scene,” EYE, vol. 3,p. P3, 2001.

[93] F. Biocca, A. Tang, C. Owen, and F. Xiao, “Attention Funnel:Omnidirectional 3D Cursor for Mobile Augmented RealityPlatforms,” in Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, ser. CHI ’06. New York,NY, USA: ACM, 2006, pp. 1115–1122. [Online]. Available:http://doi.acm.org/10.1145/1124772.1124939

[94] E. Veas, E. Mendez, S. Feiner, and D. Schmalstieg, “Direct-ing Attention and Influencing Memory with Visual SaliencyModulation,” in Proc. of the International Conference on HumanFactors in Computing Systems, 2011.

[95] N. A. M. ElSayed, R. T. Smith, and B. H. Thomas, “Horuseye: See the invisible bird and snake vision for augmentedreality information visualization,” in 2016 IEEE InternationalSymposium on Mixed and Augmented Reality (ISMAR-Adjunct).IEEE, 2016, pp. 203–208.

[96] S. Sakai, Y. Yanase, Y. Matayoshi, and M. Inami, “D-ball:virtualized sports in diminished reality,” in Proceedings of theFirst Superhuman Sports Design Challenge on First International

Symposium on Amplifying Capabilities and Competing inMixed Realities - SHS ’18. New York, New York, USA:ACM Press, 2018, pp. 1–6. [Online]. Available: http://dl.acm.org/citation.cfm?doid=3210299.3210305

[97] S. K. Feiner and D. Duncan Seligmann, “Cutaways AndGhosting: Satisfying Visibility Constraints In Dynamic 3dIllustrations,” The Visual Computer, vol. 8, pp. 292–302, 1992.

[98] A. Dey, A. Cunningham, and C. Sandor, “Evaluating depthperception of photorealistic mixed reality visualizationsfor occluded objects in outdoor environments,” in ACMSymposium on Virtual Reality, 2010. [Online]. Available:http://dl.acm.org/citation.cfm?id=1889911

[99] S. Ghasemi, M. Otsuki, and P. Milgram, “Use of randomdot pattern for achieving x-ray vision with stereoscopic aug-mented reality displays,” in 2016 IEEE International Sympo-sium on Mixed and Augmented Reality (ISMAR-Adjunct), Sep.2016, pp. 250–251.

[100] T. Strothotte and B. Preim, Abstraction in interactive computa-tional visualization. Springer Berlin, 1998.

[101] “Interactive taxonomy.” [Online]. Available: https://stefanie.cspages.otago.ac.nz/arvispipeline/

[102] “Interactive grouping of taxonomy.” [Online]. Avail-able: https://stefanie.cspages.otago.ac.nz/arvispipeline/parallelset.html

[103] R. Grasset, A. Duenser, H. Seichter, and M. Billinghurst, “Themixed reality book,” in CHI ’07 extended abstracts on Humanfactors in computing systems - CHI ’07. New York, New York,USA: ACM Press, Apr. 2007, p. 1953. [Online]. Available:http://dl.acm.org/citation.cfm?id=1240866.1240931

[104] S. Feiner, B. MacIntyre, T. Hollerer, and A. Webster, “Atouring machine: prototyping 3D mobile augmented realitysystems for exploring the urban environment,” in Digest ofPapers. First International Symposium on Wearable Computers.IEEE Comput. Soc, 1997, pp. 74–81. [Online]. Available: http://ieeexplore.ieee.org/xpl/freeabs all.jsp?arnumber=629922

[105] D. Kalkofen, E. Mendez, and D. Schmalstieg, “Comprehensi-ble Visualization for Augmented Reality,” IEEE Transactionson Visualization and Computer Graphics, vol. 15, no. 2, pp. 193–204, 2009.

[106] C. Bichlmeier, M. Kipot, S. Holdstock, S. M. Heining, E. Euler,and N. Navab, “A Practical Approach for IntraoperativeContextual In-Situ Visualization,” in AMIARCS ’09, 2009.

[107] J. Fischer, H. Regenbrecht, and G. Baratoff, “Detecting dy-namic occlusion in front of static backgrounds for ar scenes,”in Proceedings of the workshop on Virtual environments 2003,2003, pp. 153–161.

[108] J. Fischer, B. Huhle, and A. Schilling, “Using time-of-flight range data for occlusion handling in augmentedreality,” in Proceedings of the 13th Eurographics conference onVirtual Environments (EGVE’07), 2007, pp. 109–116. [Online].Available: http://dl.acm.org/citation.cfm?id=2386061

[109] M. Lerotic, A. J. Chung, G. Mylonas, and G.-Z. Yang, “Pq-Space Based Non-Photorealistic Rendering for AugmentedReality,” in Proc. MICCAI ’07, 2007, pp. 102–109.

[110] E. Mendez and D. Schmalstieg, “Importance masks forrevealing occluded objects in augmented reality,” inProceedings of the ACM symposium on Virtual reality softwareand technology - VRST ’09. New York, New York,USA: ACM Press, 2009, pp. 247—-248. [Online]. Available:http://portal.acm.org/citation.cfm?id=1643928.1643988

[111] S. Zollmann and G. Reitmayr, “Dense depth maps fromsparse models and image coherence for augmented reality,”in Proceedings of the 18th ACM symposium on Virtual reality soft-ware and technology, Dec. 2012, pp. 53–60. [Online]. Available:http://dl.acm.org/citation.cfm?id=2407336.2407347

[112] M. A. Livingston, J. E. S. II, J. L. Gabbard, T. H. Hollerer,D. Hix, S. J. Julier, Y. Baillot, and D. Brown, “ResolvingMultiple Occluded Layers in Augmented Reality,” Symposiumon Mixed and Augmented Reality, 2003. [Online]. Available:http://portal.acm.org/citation.cfm?id=946796

[113] S. Zollmann, G. Schall, S. Junghanns, and G. Reitmayr,“Comprehensible and Interactive Visualizations of GIS Datain Augmented Reality,” Advances in Visual Computing, pp.675–685, 2012. [Online]. Available: http://link.springer.com/chapter/10.1007/978-3-642-33179-4 64

[114] E. Mendez, “Visualization On the Usage of Context for


http://doi.acm.org/10.1145/1518701.1518871

http://doi.acm.org/10.1145/1518701.1518871

http://doi.acm.org/10.1145/1851600.1851655

http://doi.acm.org/10.1145/2742647.2742666

http://doi.acm.org/10.1145/2661229.2661256

http://doi.acm.org/10.1145/3132129.3132144

http://doi.acm.org/10.1145/3229434.3229438

http://doi.acm.org/10.1145/3229434.3229438

http://doi.acm.org/10.1145/1124772.1124939

http://dl.acm.org/citation.cfm?doid=3210299.3210305

http://dl.acm.org/citation.cfm?doid=3210299.3210305


https://stefanie.cspages.otago.ac.nz/arvispipeline/

https://stefanie.cspages.otago.ac.nz/arvispipeline/

https://stefanie.cspages.otago.ac.nz/arvispipeline/parallelset.html

https://stefanie.cspages.otago.ac.nz/arvispipeline/parallelset.html


http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=629922

http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=629922




http://portal.acm.org/citation.cfm?id=946796

http://link.springer.com/chapter/10.1007/978-3-642-33179-4_64

http://link.springer.com/chapter/10.1007/978-3-642-33179-4_64


Augmented Reality,” Ph.D. dissertation, Graz University ofTechnology, 2010.

[115] J. Looser, M. Billinghurst, and A. Cockburn, “Through thelooking glass: the use of lenses as an interface tool forAugmented Reality interfaces,” in Computer Graphics andInteractive Techniques in Australasia and South East Asia, 2004.[Online]. Available: http://portal.acm.org/citation.cfm?id=988870

[116] R. Spence and M. Apperley, Bifocal Display. Aarhus,Denmark: The Interaction Design Foundation, 2011.[Online]. Available: http://www.interaction-design.org/encyclopedia/bifocal display.html

[117] C. Sandor, A. Cunningham, U. Eck, D. Urquhart,G. Jarvis, A. Dey, S. Barbier, M. R. Marner, andS. Rhee, “Egocentric space-distorting visualizations forrapid environment exploration in mobile mixed reality,” inIEEE International Symposium on Mixed and AugmentedReality (ISMAR 2009). Ieee, Oct. 2009, pp. 211–212. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5336461

[118] R. Azuma and C. Furmanski, “Evaluating label placementfor augmented reality view management,” The SecondIEEE and ACM International Symposium on Mixedand Augmented Reality, 2003. Proceedings., pp. 66–75,2006. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1240689

[119] E. Rosten, G. Reitmayr, and T. Drummond, “Real-timevideo annotations for augmented reality,” Advances in VisualComputing, 2005. [Online]. Available: http://link.springer.com/chapter/10.1007/11595755 36

[120] N. Norouzi, G. Bruder, B. Belna, S. Mutter, D. Turgut, andG. Welch, “A systematic review of the convergence of aug-mented reality, intelligent virtual agents, and the internet ofthings,” in Artificial Intelligence in IoT. Springer, 2019, pp.1–24.

Stefanie Zollmann Stefanie Zollmann is aSenior Lecturer at the University of Otagoin New Zealand. Before joining the Univer-sity of Otago, she was a developer and re-searcher working with Animation ResearchLtd. Before that, she worked as a postdoc-toral researcher at the Institute for ComputerGraphics and Vision at Graz University ofTechnology. In 2007, she graduated in MediaSystems at Bauhaus University Weimar, Ger-many and 2013, she obtained a PhD degree

from the Institute for Computer Graphics and Vision in Graz. Hermain research interests are visualization techniques for augmentedreality and entertainment, but also include mobile augmented realityand spatial augmented reality.

Tobias Langlotz (Member IEEE) is an As-sociate Professor at the University of Otago.Tobias was previously a senior researcherat the Institute for Computer Graphics andVision (Graz University of Technology, Aus-tria) where he also obtained his PhD. Tobiasmain research interest is Vision Augmen-tations and Computational Glasses utilizingAR technology, spontaneous interaction forwearable AR systems, and nomadic mobiletelepresence solutions, where he works at

the intersection of HCI, Computer Graphics, Computer Vision andUbiquitous Computing.

Raphael Grasset Raphael Grasset is a tech-nical fellow at PTC. He was previously princi-pal engineer at PTC (Nov 2015-June 2018),staff engineer at Qualcomm Research Aus-tria (Nov 2013- Nov 2015), senior researcher(2011-2014) at the Institute for ComputerGraphics and Vision (TU-Graz, Austria) andaffiliated researcher at the Human InterfaceTechnology Laboratory New Zealand (Uni-versity of Canterbury, New Zealand). Be-tween 2004 to 2010, he was a senior re-

searcher (2007-2010) and postdoctorate at the HIT Lab NZ (2004-2007). He completed his PhD in Computer Science at ARTIS (ex-iMAGIS Lab, INRIA, Universit Joseph Fourier, France) in 2004. Hismain research and developments interests include: 3D interaction,computer-human interaction, augmented reality, mixed reality, visu-alization and CSCW.

Wei Hong Lo is currently a PhD studentat the University of Otago, majoring in thefield of Augmented Reality Sports Visualiza-tion and Human-Computer Interaction (HCI).Prior to his PhD, he graduated from theUniversity of Nottingham Malaysia Campusin Software Engineering in 2017, where heobtain interest in the field of HCI and visual-ization. He then proceeded as a research as-sistant and was involved in projects such asdeveloping interactive exhibits for museums.

Shohei Mori is a University Project Assistantat the Institute of Computer Graphics (ICG)and Vision at Graz University of Technol-ogy. His focus lies in augmented and dimin-ished reality and the related computer visionand display technologies. He received hisB.S., M.S., and PhD in engineering at Rit-sumeikan University, Japan, in 2011, 2013,and 2016, respectively. Before joining theICG, he worked as a Research Fellowshipfor Young Scientists (DC-1) from the JSPS

during his Ph.D degree, and he started his career as a JSPSResearch Fellowship for Young Scientists (PD) at Keio University,Japan.

Holger Regenbrecht (Member IEEE) is aProfessor at the University of Otago in NewZealand. He received his doctoral degreefrom Bauhaus University Weimar, Germany.Holger’s general field of research is Human-Computer Interaction with an emphasis onvisual computing, in particular virtual andaugmented reality. His work spans theory,concepts, techniques, technologies, and ap-plications in a wide range of domains. A sec-ond emphasis of his research is computer-

mediated communication, such as telepresence systems, wherehe studies technological and psychological aspects and deliversprototype solutions.



http://www.interaction-design.org/encyclopedia/bifocal_display.html

http://www.interaction-design.org/encyclopedia/bifocal_display.html





http://link.springer.com/chapter/10.1007/11595755_36

http://link.springer.com/chapter/10.1007/11595755_36

Documents

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER … · Reality: A Taxonomy, Methods and Patterns Stefanie Zollmann, Tobias Langlotz, Raphael Grasset, Wei Hong Lo, Shohei Mori and