Spatially Aware Mobile Interface for 3DVisualization and Interactive Surgery Planning

Jeronimo G. Grandiand Anderson Maciel

Federal University of Rio Grande do SulUFRGS - Porto Alegre, Brazil{jggrandi,amaciel}@inf.ufrgs.br

Henrique G. DebarbaEcole Polytechnique Federale de Lausanne, EPFL

Lausanne, Switzerlandhenrique.debarba@epfl.ch

Dinamar J. ZanchetSurgery Department, UFCSPA

Porto Alegre, Brazildjzanchet@gmail.com

Abstract—While medical images are fundamental in thesurgery planning procedure, the process of analysis of suchimages slice-by-slice is still tedious and inefficient. In this workwe introduce a system for exploration of the internal anatomystructures directly on the surface of the real body using amobile display device as a window to the interior of thepatient’s body. The method is based on volume visualizationof standard computed tomography datasets and augmentedreality for interactive visualization of the generated volume. Itsupports our liver surgery planner method in the analysis ofthe segmented liver and in the color classification of the vessels.We present a set of experiments showing the system’s ability tooperate on mobile devices. Quantitative performance results aredetailed, and applications in teaching anatomy and doctor-patientcommunication are discussed.

I. INTRODUCTION

The main contribution of this paper is an augmentedreality-based mobile display interface as depicted in theschematic of Fig. 1. It is a significant contribution as it makespossible the integration of volume visualization of real medicalimages with spatial information about the user and the data.Such interface has direct applications in surgery planning,teaching anatomy and doctor-patient communication. In thispaper we present an application in anatomic hepatectomyplanning.

Anatomic hepatectomies [1] (anatomic liver resections) areprocedures used for treatment of liver tumor patients andin living related liver transplantation. Planning hepatectomiesinvolves, in addition to medical knowledge, a computationalprocedure which allows boundary determination of the liveron tomographic images. This process is called segmentation,and it is also necessary to calculate liver volume. Althoughliver segmentation in computed tomography (CT) has alreadymotivated a number of research works, its practical applica-tion still represents a major challenge. Most of the medicalimages segmentation techniques are based on algorithms ofregion growing and active contours, which are computationallydemanding, inaccurate or both. Besides organ segmentation,accurate and fast classification of the liver functional segmentsis crucial in liver tumor extraction as this procedure is signifi-cantly influenced by the estimation of the volume of the liverand its anatomical subdivisions. This is a major issue becausethe estimated percentage of remaining liver tissue may affectthe decisions made in surgery planning and even make theprocedure inviable [2].

Fig. 1. Conceptualization of our interface. One or more mobile displays areaimed at the patient’s body and the observers holding them can see the internalstructures of the body in focus. The system can run on multiple devices at thesame time providing different angles of observation depending on the positionand orientation of the observer in the real environment. Furthermore, thesystem wirelessly communicates, allowing users to share information amongthem. Information exchanged is points of interest and it can be marked directlyon the volume through the device’s touch screen.

Currently, two usual visualization approaches are availablefor hepatectomy and other surgery planning: direct analysisslice-by-slice (Fig. 2a) and volume visualization (Fig. 2b).Both are non-efficient and require an experienced professionalto interpret the images, as they are coarse abstractions whichdo not map intuitively to the anatomy they represent. Atthe same time, anatomy students spend many years beforethey acquire the ability to think in 3D using only 2D data.Expensive cadavers and anatomy mannequins are the only 3Dtools available for them to construct a mental model of theinterior of the body in association with the landmarks on thebody surface. Yet, many patients have difficulty to understandtheir disease from the slice data shown by their physicians.

Meanwhile, natural interfaces exploiting speech, gesturesand presence in the virtual environment are replacing the cur-rently conventional mouse and keyboard interaction paradigms.The miniaturization of computer devices and the increase intheir processing power allowed smaller equipments to performcomplex tasks which were once only possible on dedicatedworkstations. This also allowed such a level of mobility that awhole new set of applications, which were unthinkable withoutthose resources, are now spreading fast.

Fig. 2. Previous approaches: (a) the process of analyzing images slice-by-sliceis tedious and inefficient; (b) the use of expensive workstations for volumevisualization is non-intuitive and requires intensive training.

In this context, a system allowing the physician to visu-alize the inner structures of a patient’s body, using a mobiledisplay as a see-through device [3] with semi transparency andopacity, is desirable. Such system would make visualizationand exploration of the inner anatomy a more intuitive andnatural procedure when compared to traditional visualizationmethods. Moreover, while head-mounted displays (HMDs),see-through glasses and other devices are expensive, the use ofa mobile screen device, such as tablets and smartphones offersa practical, low cost and widely available solution.

In the present work, we introduce an approach to volumevisualization carried out through the mobile display interface.Such human-computer interface is based on augmented realityfor interactive visualization of the volume generated from CT.The system allows for the analysis of inner body structuresby pointing the mobile display, aware of its position in space,directly upon the body. The display functions as a windowto the interior of the body. It serves as a tool for teachinganatomy, for facilitating doctor-patient communication and itsupports our liver surgery planning method described in aprevious work [4].

II. BACKGROUND AND RELATED WORK

Scientific visualization techniques are proving to be usefulin a number of applications in the medical field, including di-agnosis, surgery simulation and planning, and medical training[5]. Volumetric medical images visualization is a focus in thispaper. Although medical professionals can produce accuratediagnostics based only on the raw data [6], there is a greatinterest in visualizing these data in 3D, especially complexorgans, like the liver and its vascular system. 3D visualizationallows for a shorter learning curve for anatomy students andalso for a reduced rate of interpretation error among lessexperienced personnel [7].

Along the years, a number of volume rendering techniqueshave been developed. They can be split in two main classes[8]: indirect volume rendering, which extracts the surfaces andrepresent them by means of graphic primitives; and directvolume rendering, which generates the image directly fromthe volume data. Among indirect rendering, we cite MarchingCubes [9] and Contour-connecting [10]. In direct volumerendering, ray tracing [11], splatting [12] and texture mappingare the most common approaches. In this work we use a 2Dtexture mapping GPU accelerated volume rendering algorithm

based on the pre-integrated volume rendering using hardware-accelerated pixel shading by Engel et al. [13].

Virtual reality (VR) is widely known as a computer sim-ulated world in which a user can navigate in real time asif they were moving in a real three-dimensional space andfeel present in that space [14]. Augmented reality (AR) isstill misunderstood in multidisciplinary areas. Paul Milgramcreated the virtuality continuum, which is an axis where thereal world is in one side and the virtual world (VR) inthe other side. Between them, the continuum is genericallydefined as mixed reality, and represents an environment inwhich the objects from the real and the virtual worlds arepresented together in the same context [15]. Yet, in thecontinuum, mixed reality is subdivided into augmented reality(AR) and augmented virtuality. AR can also be defined asan improvement of the real world with computer generatedelements, or a system which supplements the real world withvirtually generated objects coexisting in the same space [16].

Augmented reality requires capturing the real world to thecomputer system (computer vision). This is usually made usinga video camera or a webcam. The basic requirements in theconstruction of an AR system are tracking, registration anddisplay [17].

In tracking, the first step is capturing a video frame. Theframe is then analyzed and patterns are recognized, whichcan be markers (tags) or pre-established information (likecontours, edges and an image pattern). When these patternsare found, the system registers their positions and calculatesthe transformation of the pattern on the frame in relationto the camera. Then, the three-dimensional virtual object isassociated to the pattern according to the estimated pose(position and orientation) registered for the pattern and canbe displayed at that emplacement. Tracking, registration anddisplay procedures are repeated continuously during the videocapture.

In this work we used the Vuforia library that can recognizeand track previously learned patterns in the video imagecaptured from the real environment. The library automaticallycalculates the transformations needed to place a virtual objectin the right pose according to the position and orientation of thepattern. This allowed us to prove the concept of our interface.The use of a known pattern has demonstrated great practicalresults and is fast enough for real time applications, especiallyon mobile devices with limited computational power. Forreliable tracking and pose estimation, the patterns are imageswith easy to extract visual features. They are printed in paperor other material, like clothing, and do not need any electronicdevice associated.

We presented limited versions of our interface with tabletPCs [18] and an iPad [19] recently as a proof of concept. How-ever, the use of tablets in medial interventions is new and theliterature in the field is not abundant. In 2013, Karl Oldhafer,a surgeon in Hamburg used a tablet in the operating roomto demonstrate a few applications developed by researchersfrom the Fraunhofer Institute. They used the tablet’s integratedcamera to film the liver during the operation. The app wasable to superimpose some planning data, including vesselsin different colors. However, the user was required to moveand turn the device until they manually aligned, with some

degree of accuracy, the real liver video with the planningdata. Despite the media repercussion, neither Oldhafer norFraunhofer published any results so far.

III. DEVELOPMENT OF AN INTERACTIVE SPATIALLYAWARE VISUALIZATION SYSTEM

This section describes the interactive spatially aware vi-sualization system that gives support to the liver segmentsclassification tool based on vessel branching described in aforthcoming section.

A. System Design

The system has been conceived to be used with tabletcomputers. There are essentially two classes of tablets com-mercially available. The first is a special class of laptops whichallow rotating the screen and closing the lid with the displayturned upwards. The second are pure tablets, with no lid atall, and with a multi-touch display spreading through almostentirely one of its sides, as the Apple iPadTM , the SamsungGalaxy TabTM and many others. They all allow interactiondirectly on the screen with one or more fingers or a stylus,without the need of mouse or keyboard. Most of them areequipped with one or two embedded video cameras.

The system allows the visualization of internal structuresof the body using a tablet as a mobile see-through display.With the aid of the built-in camera, the display shows a realtime image of the real environment behind it as if it were awindow to the world. At the same time, the system generatesa 3D visualization of the CT data which is mixed with the realworld video on the display, augmenting the reality. In practice,after a printed pattern is conveniently placed on a specificlandmark of the patient’s body, AR permits the association,in a common space, of medical image volumes with the realbody from which they were taken. The effect is a penetratingvision that sees through the patient’s skin directly into thepatient’s body, as illustrated in Fig. 3.

The volume viewer has been implemented based on thepaper by Engel, Kraus and Ertl [13], as previously mentioned.The choice for this kind of viewer was based on: highframerates on common hardware; good quality with few slices(as few as 40); possibility to change the window parametersfor the transfer functions. The volume is represented by voxels(volume pixels). The original implementation takes a set of 3Dtextures as input and was aimed for PC platform. This typeof data structure is natively supported in current versions ofOpenGL, but in the version for mobile systems, known asOpenGL ES, this feature is only implemented as an extension,leaving the hardware manufacturer the choice to adopt suchfeature. As three-dimensional texture is not a standard, limitingthe range of supported devices, we modified the algorithm toload and manipulate two-dimensional textures.

The implementation uses C++ language, OpenGL ES andGLSL shader language. The volume viewer allows changingthe camera angle and position, and changing the center andwidth of the density visualization range – also called window– for viewing different materials (tissue densities) in the bodyvolume. The control of the density window parameters occursdirectly on the screen, through touch and drag.

Fig. 3. Actual photograph of our tablet-based interface for 3D medicalvisualization. In this example, the patient is wearing a t-shirt with the printedpattern.

Also, to adapt with different hardware profiles, we addedthe possibility to change the volume shading quality in realtime. This allows cuts in the algorithm steps, reducing thecomplexity in order to increase the system performance. Thereare two main GPU intensive calculations in the algorithm, thez-axis linear interpolation between slices and the pixel colorclassification based on a pre-integration table. To allow users tochose the suitable balance between quality and performance ofthe rendering for the available hardware, we included two but-tons in the user interface. These buttons act as on/off switchesactivating or deactivating the interpolation and pre-integrationcalculation. This allow some control on the compromise offaster refresh rates with lower quality, or better visual qualitywith lower refresh rates. In Fig. 7 we show the appearance ofdifferent shading qualities.

The Vuforia library manages the AR interaction with theOpenGL ES virtual camera, it allows the volume data to bevisualized according to the position and orientation of the userhandling the mobile display and the pattern tracked by thetablet’s video camera in real time. While the back camera isused in the doctor-view-patient scenario, the front camera ofthe device can be used for self-view, analogous to a mirror,in a doctor-patient communication scenario where the patienthas to visualise his/her own body.

B. Collaborative use

Our system is not limited to one display at a time. Usershandling a display device connected through wireless networkcan observe the data from their own point of view aroundthe real patient. See Fig.1 for an illustration. The system alsoallows information exchange among the several users. One

Fig. 4. Surgery planning pipeline: (a) liver is segmented using LiverSegmentstool; (b) the segmented liver is obtained from SmartContour, the densitywindow is ajusted to display blood vessels upon which the user definespoints; (c) liver classified according to Couinaud segments; (d) 3D volumevisualization allows users to choose which structures to display and when; (e)using the mobile display to see the 3D volume directly on the surface of adummy’s body.

example is to select/create points of interest on the anatomyby pointing on the display with the cursor/finger. Such interestpoints are shown to all users connected on the collaborativeplatform. The interest points are placed directly in the volumedata according with the position of the voxel.

Sharing interest points can be useful, for example, inanatomy teaching. The teacher can identify a given structureon the data and place a marker to share this information withthe students. Every student will see the marker on their displayin the precise 3D location. Many other parameters can also beshared depending on the final application.

In our implementation, collaboration is achieved by ex-changing messages through all devices running the application.We used a simple client-server approach to prove the concept,with the server running on one of the tablets and the client onall of them.

IV. SURGERY PLANNING

In this section we present the decision flow pipeline forplanning hepatectomies and our surgery planner tools thatsimplify the process and give to the doctor more intuitive-ness to make decisions. The pipeline consists of three mainsteps performed after the acquisition of the images (datasets):

image segmentation, vessel classification, and identification ofdamaged areas of the liver, as shown in Fig. 4.

Diagnosis and planning start with the CT acquisition.Conventional contrast injections are used, and the three phasesof the liver circulation (portal, venous and arterial) can beacquired separately over time, as usual. They can also beacquired all together after a sufficient amount of contrast isperfused through the 3 systems. In either case, datasets are thenexported in DICOM format from the CT scanner. They can beimported into the image segmentation tool (SmartContour) forliver segmentation.

SmartContour semi-automatically segments the CT dataset.The tool has been designed to require the minimum user inter-vention. It combines automatic edge detection with parametriccurves to quickly create organic contours of the segmentedareas. Afterward, both the original images and a 3D segmen-tation mask are exported. This information serves as input forthe vessel classification tool, LiverSegments.

In LiverSegments the user (radiologist or surgeon) inspectsthe 3D volume of the liver, showing or not the surroundingorgans. At this point, the total volume of the liver is alsocalculated and displayed. Then, interactively, the user sets thecenter and threshold of the density window for visualization.Selected density ranges are set to transparent allowing thevessels to be highlighted (Fig. 4b). At this moment the user isable to classify the liver segments (for example, Couinaud’s)by clicking/tapping on vessel branches directly on the 3D view.Such branch selection eventually produces a color distributionnot only on the vessels but also in the neighboring regionsof the parenchyma (Fig. 4c and Fig. 4d). Each colored regioncorresponds to one functional segment. Borders between theseregions are suggested as incision lines for surgery. Whensegments are selected for removal, the functional volumeof the remaining liver is also calculated and displayed forconsideration.

After patient specific liver segmentation and segment clas-sification processes are done with the programs describedabove, damaged areas can be identified in 3D using thespatially aware AR interface (Fig. 4e) and the physician candecide which segments will be removed surgically. They canalso plan how to make the incisions in such a way thatthey avoid unnecessarily sectioning of important vessels. Otherimportant information that can be obtained is related to totaland functional volumes. Total liver volume and each functionalsegment volume are automatically calculated by the LiverSeg-ments tool. These volumes, together with the identification ofpotential areas of ischemia and venous stasis in consequenceof surgery, are taken into account to estimate the total volumeof the remaining functional tissue.

V. SYSTEM ASSESSMENT

We propose a quantitative experiment to evaluate the sys-tem performance in different hardware configurations (scala-bility). The system performance is measured in millisecondsto render a frame, that can be easily converted to frames persecond (FPS), which means the number of frames registered,processed and displayed by the device in every second. Theframe rate which generates smooth and continuous transactionsin computer graphics is around 30 fps (approx. 33ms). At

Tablet specsConfig. iPad3 iPad2

CPU Dual Cortex A9 1Ghz Cortex A9 1GhzRAM 1GB 512MBGPU PowerVR 543MP4 PowerVR 543MP2

Screen Res. 2048x1536 1024x768

Smartphone specsiPod4/iPhone4 GalaxyS

CPU Cortex A8 900Mhz Cortex A8 1GhzRAM 256MB 512MBGPU PowerVR 535 PowerVR 540

Screen Res. 640x960 480x800GalaxyS3Mini GalaxyS4

CPU Dual Cortex A9 1Ghz Quad Core Krait 400RAM 1GB 2GBGPU Mali 400 Adreno 320

Screen Res. 480x800 1080x1920TABLE I. HARDWARE CONFIGURATION USED IN THE EXPERIMENTS.

BESIDES SELECTING TABLET AND SMARTPHONE MODELS FROMDIFFERENT BUILDERS, WE ALSO CHOSE CONFIGURATIONS WITH

DIFFERENT CPU AND GPU ARCHITECTURES.

this rate or higher, the perception of continuous motion andinteraction is guaranteed. Too much below that, the continuityis lost and it frustrates the user. The framerate is proportionalto the processing power of the hardware. A machine withhigh processing power (CPU + GPU) usually provides a highaverage of FPS.

A. Experimental setup

a) Hardware: Different general purpose hardware pro-files have been used. Table I shows the devices, which includestablets and smartphones. Moreover, we cared to select differentCPU and GPU architectures to assess specific advantages anyarchitecture might present.

b) Volume dataset: Two conventional CT datasets wereused for all experiments. The volume characteristics are shownin Fig. 5. The Thorax has 128x128x64 resolution and consti-tutes the lower portion of the thorax, from the base of the lungsto the kidneys, therefore including the liver. The Face dataset256x256x256 resolution and constitutes the lower portion ofthe head, from the nose to the maxillary.

Fig. 5. The two datasets used in tests. Both were acquired from conventionalCT. Thorax comprehends the base of the lungs to the kidneys and the Facecomprehends the mandible and the maxillary.

DatasetsThorax Face

Resolution 128x128x64 256x256x256TABLE II. THE RESOLUTION OF THE TWO 3D DATASETS.

Fig. 6. The two camera distances used in the tests. In this case the volumeis shown in a top view. The dark gray rectangle represents the bounding boxwhere the internal pixels are taken into account during the rendering process.In image (a), approximately 12% of the pixels represent the volume and it isthe farthest distance used for test purposes, while in (b) approximately 50%of the pixels represent the volume; this is the closest distance. The imageswere acquired at a resolution of 1024x768 on an iPad2.

B. Test protocol

To ensure uniformity of the data collected during the tests, aprotocol was stipulated. The protocol defines some parametersto be used by all devices during the tests. The parametersare applied to the virtual camera, viewport resolution, transferfunction and shader quality.

c) Virtual camera: Normally the developed systemworks with a dynamic virtual camera that change its positionand orientation according to the device position. But for bettercontrol during the tests, the virtual camera was set manually atfixed position and orientation, forcing the volume to be insidethe viewport. This way, we can easily compute the number ofpainted pixels and establish its relationship with the time torender a frame. However, this approach does not include timespent processing the scene for tracking and positioning of thevirtual camera, so we decided to capture it separately. Thus,combining the rendering time with the tracking time, we obtainthe total processing time of each frame. The distances from thevirtual camera to the volume are exemplified in Fig. 6.

d) Viewport resolution: The devices chosen for testhave different native screen resolution (see rows Screen Res. inTable I). To achieve homogeneous results, we have establishedthe same viewport resolution for all devices: 480x320. Othertests with native resolution were also logged to evaluate per-formance loss when compared with lower resolution viewport.

e) Transfer function: To display different tissue densi-ties, the system allows interactively change the transfer func-tion values. However, for the sake of experimental evaluation,we fixed this parameter. The delay is minimum and does notaffect the system performance as it only occurs during user

interaction, which does not occur in this experiment.

f) Shader quality: For the experiment, we have alsodiscretized the shader qualities in three types: low, mediumand high. These qualities are related to the description of thevolume algorithm presented in section III-A. Fig. 7 a,b and cdisplay the respective qualities.

Fig. 7. The three volume rendering qualities. In sequence, (a) low, (b) mediumand (c) high quality. Low quality has no improvement. Medium has the pixelcolor classification based on the pre-integration table and high also includesthe z-axis slice interpolation.

The system automatically saves logs containing the testsetup and the time in milliseconds to render each frame.

C. Performance Results

g) Performance at different distances: In this experi-ment we analyze the system performance according to thenumber of pixels which the color depends on the volume.The test aims at checking if such number interferes with thesystem performance. For this experiment we tested all deviceswith 480x320 viewport resolution loading the Thorax dataset(Fig. 5 Thorax). The test consists in measuring the time (inmilliseconds) to render each frame at different display-subjectdistances. Distance specifications and their relation with thenumber of pixels can be seen in Fig. 6. The closer to the subjectthe camera is, the greater the number of color dependent pixels.The chart of Fig. 8 illustrates the results. This test revealsthat the system is bound by the GPU and the amount ofpainted pixels on screen that represents the volume interferessignificantly in the processing speed of each frame. This is dueto the approach used by the shader that renders the volume.At the closest virtual camera distance, approximately 40%of painted pixels represent the volume, while at the farthestdistance only about 10% belong to the volume.

h) Performance with different devices: The objective ofthis test is to check the performance capabilities of currentavailable tablets. Additionally, this test shows how suitablethe system developed is to run in smaller mobile devices,such as smartphones. In the worst case (native resolution andhigh quality rendering), less powerful smartphones (GalaxySand SIII Mini) do have performance lower than acceptable.On the other hand, the iPad3 and the GalaxyS4, achievedsmooth frame sequences. In lower graphics settings, 480x320resolution and low quality rendering the performance with alldevices is acceptable. The chart in Fig. 9 presents results forthe average case.

i) Performance with different datasets: Datasets canbe very heterogeneous depending on the medical scanner andbody part captured. They can vary from the number of slicessampled to the low-level type and arrangement of captureddata values (8-bits, 16-bits). To address such variety, we have

0ms

30ms

60ms

90ms

120ms

150ms

180ms

210ms

iPad3 iPad2 iPod4 GalaxyS4 GalaxyS3Mini GalaxyS

Tim

e in

mill

isec

onds

Devices

Aprox. 50.6% volume pixelsAprox. 12.2% volume pixels

Fig. 8. Performance in different distances. Both virtual camera distancesshown in Fig. 6 were used. The test was made with all devices in 480x320viewport resolution and medium rendering quality.

0ms

30ms

60ms

90ms

120ms

150ms

180ms

210ms

iPad3 iPad2 iPod4 GalaxyS4 GalaxyS3Mini GalaxyS

Tim

e in

mill

isec

onds

Devices

Fig. 9. This graph represents the performance in an average use case. Thevertical axis presents the time necessary to compute 1 frame; the lesser thebetter. Test setup: all devices in 480x320 viewport resolution and mediumrendering quality.

measured the system performance with two distinct datasets(see Fig. 5 and Table II for more details). The Thorax datasethas fewer slices then the Face, but is larger in size (megabytes).

We then decided to eliminate this variable from the exper-iment. For evaluation proposes we fixed the number of slicesand resolution to be rendered in the algorithm. For this reason,we observed no performance difference between datasets eventhough they have size, texture resolution and number of slicesthat differ. The dataset size only affects the main memoryload. It can take a few seconds to load a large dataset, butafter that, we did not want the size to affect the renderingperformance. In other words, even if the texture resolutionincreases the size of the dataset, our algorithms map the imagesonto fixed resolution texture planes. If the texture is smaller,the pixels are stretched, and they are merged if the textureis larger. The number of dataset slices could also also affectthe rendering time if all slices were rendered. However, thenumber of slices to be rendered was set to a fixed value in theshader configuration.

j) Performance in different viewport resolutions: Thedisplays of our mobile devices have a high pixel density,e.g., iPod 4 has 326 ppi (pixels per inch) and GalaxyS4 has441ppi. Such high resolution screens generate clear and sharpimages, ideal to display text and photos. However, it can bea drawback for GPU intensive calculations when rendering

0ms

30ms

60ms

90ms

120ms

150ms

180ms

480x320 2.048x1.536

Tim

e in

mill

isec

onds

Resolution

Aprox. 50.6% volume pixelsAprox. 12.2% volume pixels

Fig. 10. In this test the iPad3 was used to compare the performance indifferent viewport resolution. The native resolution (2048x1536) is comparedwith the lower resolution (480x320) rendering the volume in medium quality.

0ms 30ms 60ms 90ms

120ms 150ms 180ms 210ms 240ms 270ms 300ms 330ms 360ms 390ms 420ms 450ms 480ms 510ms 540ms 570ms 600ms 630ms

iPad3 iPad2 iPod4 GalaxyS4 GalaxyS3Mini GalaxyS

Tim

e in

mill

isec

onds

Resolution

Low QualityMedium QualityHigh Quality

Fig. 11. Time versus devices for three volume shading qualities. Usedsettings: 50% of the pixels painted, all devices, low resolution viewport(480x320). As observed, the iPad3 and GalaxyS4 achive near to 30ms perframe (approx. 30 FPS) in low and medium resolution. This combinationdelivers the best user experience. Other devices only achive regurar experiencein low quality shader.

3D volumes in real time, especially to larger tablet screens.We have used the iPad3 to compare the performance of ourvolume rendering with native resolution (2048x1536) againsta lower resolution (480x320). Comparative results are shownin the chart of Fig. 10. Notice that increasing screen resolutionis greatly detrimental to performance. On the other hand,lowering the viewport size and stretching the image to fill thescreen increases the pixel size and worsens the image quality.

k) Performance with different shader qualities: As dis-cussed in subsection III-A, we have applied algorithm cutsthat simplify the render quality in order to achieve betterperformance. In the chart of Fig. 11, we compare the perfor-mances switching qualities in different devices. As expected,better shading quality delivers lower frame rate. The mediumquality (pre-integration calculation activated) balances qualityand acceptable frame rate.

VI. CONCLUSION AND FUTURE WORK

In this paper we presented our strategy and tools forliver surgery planning which takes into account the bloodvessel branching within the organ. The strategy is based onpatient specific CT datasets and computation methods for liversegmentation and segment classification.

However, our main contribution is an augmented realityinterface for interactive data visualization and preoperative

planning. On the top of our tools, such interface allows for thethorough analysis of the liver or any other body part directlyin 3D and at its actual location along with the patient’s body,which is far more comprehensive than usual methods. Thesystem allows for a novel paradigm of interactive study of thehuman anatomy. In addition to surgery planning, a system suchas this could be used to teach anatomy and pathology, and inthe doctor’s office, helping them to communicate with patientsand relatives.

Our system has been built up with off-the-shelf softwareand hardware components, which tremendously increases di-rect access to surgeons and apprentices due to the low costand wide availability. The system has been conceived to copewith the current increase of mobile and pervasive computation.In this context, all development of visualization tools hasbeen made with the advances and limitations of state-of-the-artmobile devices in mind.

Performance tests reveled that:

• our algorithms are efficient to run in tablets andsmartphones with satisfactory performance.

• the number of painted pixels influences the perfor-mance instead of the volume voxels independent ofGPU family.

• native screen resolutions with high ppi decreasessignificantly the performance mainly in low profiledevices.

• low price smartphones only obtain smooth perfor-mance with low resolution viewport and low qualityshading.

• the most suitable hardware tested is the iPad3 amongtablets and the GalaxyS4 among smartphones. Theydisplay smooth frame sequences even with high qual-ity shading and native resolution. This conclusionreinforces the argument that future hardware will notbe a bottleneck.

• shading quality can be switched in real time by userinteraction, it can provide a balance between highquality visualization when analyzing some structureand acceptable frame rates when exploring the holedata.

The tests were designed to evaluate whether the systemcould be run smoothly on common use devices. Future workwill focus on user experience to complete tasks using theproposed system.

ACKNOWLEDGMENT

The authors would like to thank the funding from CAPES,CNPq-Brazil grants #305071/2012-2 and #478730/2012-8, andFapergs grant #12/1944-2.

REFERENCES

[1] C. Couinaud, Anatomie Chirurgicale du Foie. Paris, France: Masson,1986, vol. 112, no. 5.

[2] H. Lang, A. Radtke, M. Hindennach, T. Schroeder, N. Fruhauf,M. Malago, H. Bourquain, H.-O. Peitgen, K. Oldhafer, and C. Broelsch,“Impact of virtual tumor resection and computer-assisted risk analysison operation planning and intraoperative strategy in major hepaticresection,” Archives of Surgery, vol. 140, pp. 629–638, 2005.

[3] F. Bruno, F. Caruso, L. De Napoli, and M. Muzzupappa, “Visualizationof industrial engineering data visualization of industrial engineering datain augmented reality,” J. Vis., vol. 9, no. 3, pp. 319–329, 2006.

[4] H. Debarba, D. Zanchet, D. Fracaro, A. Maciel, and A. Kalil, “Efficientliver surgery planning in 3d based on functional segment classificationand volumetric information,” in Engineering in Medicine and BiologySociety (EMBC), 2010 Annual International Conference of the IEEE,31 2010-sept. 4 2010, pp. 4797 –4800.

[5] R. Z. Tombropoulos, “Techniques and applications for tree-dimensionalvisualization in medicine.” in Proceedings of Bioengineering Confer-ence, 1999.

[6] R. A. Greenes and J. F. Brinkley, “Radiology systems,” pp. 324–365,1990.

[7] J. Marescaux, J. Clement, V. Tassetti, and et al, “Virtual reality appliedto hepatic surgery simulation: the next revolution,” in Annals of Surgery,1998, pp. 228:627–634.

[8] M. Meibner, J. Huang, D. Bartz, K. Mueller, and R. Crawfis, “Apractical evaluation of popular volume rendering algorithms,” in VVS’00: Proceedings of the 2000 IEEE symposium on Volume visualization.New York, NY, USA: ACM, 2000, pp. 81–90.

[9] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution3d surface construction algorithm,” SIGGRAPH ’87: Proceedings ofthe 14th annual conference on Computer graphics and interactivetechniques, pp. 163–169, 1987.

[10] E. Keppel, “Approximating complex surfaces by triangulation of con-tour lines.” in IBM Journal of Research and Development, 1975, p.19(1):211.

[11] M. Levoy, “Efficient ray tracing of volume data,” ACM Trans.Graph., vol. 9, no. 3, pp. 245–261, Jul. 1990. [Online]. Available:http://doi.acm.org/10.1145/78964.78965

[12] L. Westover, “Footprint evaluation for volume rendering,” j-COMP-GRAPHICS, pp. V24(4)367–376, 1990.

[13] K. Engel, M. Kraus, and T. Ertl, “High-quality pre-integrated volumerendering using hardware-accelerated pixel shading,” in HWWS ’01:Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop onGraphics hardware. New York, NY, USA: ACM, 2001, pp. 9–16.

[14] I. E. Sutherland, “A head-mounted three dimensional display,” pp. 295–302, 1998.

[15] P. Milgram, H. Takemura, A. Utsumi, and F. Kishino, “Augmentedreality: A class of displays on the reality-virtuality continuum,” inProceedings of the SPIE Conference on Telemanipulator and Telep-resence Technologies, ser. Proceedings of SPIE, vol. 2351, Boston,Massachusetts, USA, November 1995, pp. 282–292.

[16] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. Mac-Intyre, “Recent advances in augmented reality,” IEEE Comput. Graph.Appl., vol. 21, no. 6, pp. 34–47, 2001.

[17] A. D. Bowman, E. Kruijff, J. J. LaViola, and I. Poupyrev, 3D UserInterfaces Theory and Pratice. Boston, MA: Ed Addison-wesley, 2005.

[18] J. Grandi and A. Maciel, “Interactive spatially aware visualization ofmedical images,” International Journal of Computer Assisted Radiologyand Surgery, vol. 5, pp. 347–348, 2010, 10.1007/s11548-010-0471-2.[Online]. Available: http://dx.doi.org/10.1007/s11548-010-0471-2

[19] H. G. Debarba, J. Grandi, A. Maciel, and D. Zanchet, “Anatomic hepa-tectomy planning through mobile display visualization and interaction,”Studies in Health Technology and Informatics, vol. 173, pp. 111–115,2012.