TECHNICAL NOTE - BRAIN TUMORS

A novel augmented reality system of image projectionfor image-guided neurosurgery

Mehran Mahvash & Leila Besharati Tabrizi

Received: 12 December 2012 /Accepted: 18 February 2013 /Published online: 15 March 2013# Springer-Verlag Wien 2013

AbstractBackground Augmented reality systems combine virtualimages with a real environment.Objective To design and develop an augmented reality systemfor image-guided surgery of brain tumors using imageprojection.Methods A virtual image was created in two ways: (1) MRI-based 3Dmodel of the head matched with the segmented lesionof a patient usingMRIcro software (version 1.4, freeware, ChrisRorden) and (2) Digital photograph based model in which thetumor region was drawn using image-editing software. The realenvironment was simulated with a head phantom. For directprojection of the virtual image to the head phantom, a commer-cially available video projector (PicoPix 1020, Philips) wasused. The position and size of the virtual image was adjustedmanually for registration, which was performed using anatom-ical landmarks and fiducial markers position.Results An augmented reality system for image-guided neu-rosurgery using direct image projection has been designedsuccessfully and implemented in first evaluation with prom-ising results. The virtual image could be projected to thehead phantom and was registered manually. Accurate regis-tration (mean projection error: 0.3 mm) was performedusing anatomical landmarks and fiducial markers position.Conclusions The direct projection of a virtual image to thepatients head, skull, or brain surface in real time is an aug-mented reality system that can be used for image-guided

neurosurgery. In this paper, the first evaluation of the systemis presented. The encouraging first visualization results indi-cate that the presented augmented reality system might be animportant enhancement of image-guided neurosurgery.

Keywords Augmented reality system . Image projection .

Image-guided neurosurgery

Introduction

Augmented reality (AR) means systems that allow the user tosee virtual images in a real environment. The resulting imageis a combination of the real and virtual image in real time [1].Augmented reality systems are available in different types, anoptic or a video head-mounted display (HMD). In addition,head-up displays and monitor-based configurations have beendeveloped for different technical areas. Video head-mounteddisplay (HMD) systems have been described in investigationsfor improvement of image guidance [1, 3–6, 10]. The draw-back of these AR systems is the dependency on specialhardware, which can be expensive and unpractical for routineclinical application. The other point is the contradiction ofsome available systems to the definition of an augmentedreality system. Indeed, most systems use a combination of avirtual image and the video or images of the reality and not thereal environment itself.

In recent years, the use of image-guided surgery has in-creased. In neurosurgical procedures, precise preoperative plan-ning for tailored craniotomy, planning of the approach, andintraoperative image guidance of the resection extent areperformed using neuronavigation systems and are an inherentpart of neurosurgery. Visualization technologies improve theorientation and safety during the operation [7–9].

This paper describes the development of a novel methodfor image-guided surgery of brain tumor resection using aug-mented reality. We designed an augmented reality system

M. Mahvash (*)Department of Neurosurgery,Clinic of Cologne-Merheim, University of Witten-Herdecke,Ostmerheimer Strasse 200,51109 Köln, Germanye-mail: mmahvash@yahoo.de

L. Besharati TabriziMuthesius Academy of Fine Arts and Design, Industrial Design,Kiel, Germany

Acta Neurochir (2013) 155:943–947DOI 10.1007/s00701-013-1668-2

using direct image projection to combine a virtual image andthe real head, skull, or brain surface. We describe this methodand technique and present an initial evaluation performed witha head phantom, which replaces the reality environment.

Materials and methods

The augmented reality system consists of four components:

1. Virtual image creation2. Real environment3. Image projection4. Registration

Virtual image creation

A virtual image was created in two ways: (1) MRI-based 3Dmodel of the head matched with the segmented lesion of apatient and (2) a digital photograph-based model in whichthe tumor region was drawn using image-editing software.

MRI-based 3D model

T1-weighted MPR MRI datasets of a patient were used exem-plary to create a 3D model of the head and brain using MRIcrosoftware (version 1.4, freeware, Chris Rorden). The brain lesionthat was visible by contrast enhancement (gadolinium) wassegmented andmatched to the 3Dmodel with the same software(Fig. 1a). For an MRI-based brain surface model, the “brainextraction tool” (BET) was used, which segments the brainautomatically by removing the skull (“skull strip image usingBET…” from “Etc”menu). The brain lesion can be segmentedfrom the MRI dataset by definition of a “region of interest”(ROI) and saved as ROI, which can be opened (“Open ROI[s]”from “ROI” menu) to overlay it with head or brain 3D model.Different combinations are possible and the resulting 3D modelcan be rotated and viewed according to the desired perspective.

The created 3D model is a virtual image with precise localiza-tion of the tumor and can be used for image projection.

Photograph-based model

A virtual image can also be created with a digital photographof a patient’s head before or during surgery. The photographcan be used to add useful information about anatomy, tumorlocalization, and functional areas as reported previously [8].Due to the different size of the created MRI-based 3D Modelof the patient and the head phantom, for initial evaluation thevirtual image was created using a lateral photograph of thehead phantom (Fig. 2b). The tumor region was drawn in thephotograph with image-editing software. This virtual imagewas used for projection to the head phantom.

Real environment

The real environment (“surgical field”) was simulated with ahead phantom. Five fiducial markers were placed on thehead phantom as points of reference for registration of thevirtual image to the head phantom (Fig. 2a).

Image projection

For projection of the virtual image to the head phantom, acommercially available video projector (PicoPix 1020,Philips) was used based on LED technology (Fig. 1b). Thevideo projector was connected to a laptop computer with aUSB data cable. The software of the video projector wasinstalled. The video projector and the head phantom wereplaced in the same height to project the created virtual imagedirectly to the head phantom.

Registration

The position and size of the virtual image was adjusted man-ually for registration, which was performed using anatomical

Fig. 1 a Lateral view of a MRI-based 3D model of the head of apatient with a temporal brain tumor. The 3D model was created withthe MRIcro software (version 1.4, freeware, Chris Rorden). The braintumor with contrast enhancement (gadolinium) was segmented (redregion) and matched to the 3D model. b Experimental setup of the

augmented reality method including a commercially available videoprojector (PicoPix 1020, Philips) based on LED technology. Thevirtual image (laptop screen) was projected directly to the head phan-tom, which simulated the real environment

944 Acta Neurochir (2013) 155:943–947

landmarks and fiducial markers position. The registration wasperformed with complete overlapping of the five projectedfiducial markers of the virtual image and the correspondingfive fiducial markers on the head phantom (Fig. 2c, d). Theregistration was repeated five times to compare the accuracyand to evaluate the projection error after each manual regis-tration. After each registration the distance of the five fiducialmarkers (Fig. 2) to the visualized tumor border were measuredon the virtual image and on the head phantom as well.

Results

MRI-based 3D model and segmentation of the brain lesioncould be performed easily after knowing the MRIcro soft-ware well. The digital photograph-based model could becreated with visualization of tumor region. The video pro-jector (Picopix 1020, Philips) and its software could beinstalled and connected to the laptop without difficulty.

The manual registration of the virtual image and the headphantom using anatomical landmarks and fiducial markerswas possible and the tumor localization was accurate(Fig. 2d). The registration was performed within 5 min andapplied five times with the same visual accuracy to achieveprecise overlapping of the five fiducial markers from thevirtual image and head phantom. The meanmeasured distanceof each fiducial marker to the tumor border was as follows:fiducial marker 1, 32.2 mm, fiducial marker 2, 30.3 mm,fiducial marker 3, 42.1 mm, fiducial marker 4, 26.3 mm,

fiducial 5, 15.4 mm. The mean projection error was 0.3 mm(projection error range: 0.1–0.6 mm).

First evaluation results show a reliable and accurate aug-mented reality technique, which can be used for image-guided neurosurgery. The designed augmented reality sys-tem is inexpensive and easy to reproduce with a normallaptop, free available software, and a low-cost video projec-tor. The visualization results encourage testing this methodon patients in clinical investigations.

Discussion

We present a novel method of an augmented reality systemfor image-guided neurosurgery. Several augmented realitysystems have been developed for image-guided surgeryusing head-mounted displays (HMDs) [1, 3–6, 10]. Mostsystems use a combination of a virtual image and the videoor images of the reality environment and not the real envi-ronment itself. One paper presented an image overlay sys-tem using a semi-transparent display [2]. We developed theidea to project the virtual image directly to the “reality”without an HMD system or display, which can be expensiveand unpractical for clinical routine or during surgical pro-cedures. Therefore we designed a new augmented realitysystem using a video projector, which is available at lowcost. One could ask why is it inviting to design an augment-ed reality system for image-guided surgery, particularly inneurosurgery? Images that are used for navigation systems

Fig. 2 a Head phantom withfive fiducial markers (Fid.1–5). bDigital photograph-based virtualimage was created similar to thelateral view of the MRI-based3D model (Fig. 1a). The imagewas created using a lateralphotograph of the head phantomand drawing the tumor region(red region) with image-editingsoftware. This virtual image wasused for projection. c Projectionof the virtual image to the headphantom before registration. Theimage was focused, size andposition was adjusted manually.d Registration of the virtualimage to the head phantom.Anatomical landmarks and fivefiducial markers were used forregistration. Please note theprecise registration and tumorlocalization on the head surface

Acta Neurochir (2013) 155:943–947 945

are MRI and/or CT images performed preoperatively and insome cases intraoperatively. These images in different orien-tations are still virtually computed images that give neurosur-geons important information about the anatomy andlocalization of brain tumors. However, these images can bevisualized on the neuronavigation systems screen during sur-gery but are not visible before or during surgery if a neuro-surgeon looks at the head, skull, or brain surface of the patientdirectly. These are very useful images but still virtual imageswith different modalitiy and dimension as the real environ-ment and the surgeons view. Neuronavigation systems usingMRI and/or CT datasets are systems which are based onvirtual images and can be used after registration of the patient.Their disadvantage is that the surgeon must look away fromthe surgical field, look to the navigation screen and back inorder to transfer the information of the MRI and/or CT imagesin his mind from the navigation screen into the real surgicalfield. This thinking process means processing of two differentimage modalities; the MRI or CT images and the “image” ofthe real surgical environment. This is an additional work stepand can be a source of errors for surgeons that have to relatethe view of the surgical field to the different images on thenavigation monitor. It would be useful during surgery if asystem could give the surgeon access to both modalitiessimultaneously, the virtual images and the “reality”. The ideaof our augmented reality system is to integrate the helpfulinformation of the MRI and/or CT images into the real surgi-cal field to improve orientation and safety. In the describedmethod, the virtual image is projected to a head phantomdirectly without the need of additional hardware.

For the first evaluation we used a lateral photograph of thehead phantom to create the virtual image for projection due tothe different size of the patient MRI-based 3D model and thehead phantom. We could show that it is possible to create anMRI-based 3D model of head or brain easily and use it as avirtual image that can be projected to a real environment aswell (Figs. 1a, 3). The image projection can also be performed

directly to the skull or brain surface during surgery. Theplanning of skin incision and the extent of craniotomy can beimproved using this image projection technique. This systemcan be used for a “tailored” craniotomy using the imageprojection of a lesion on the patient’s skull. Furthermore,subcortical lesions that are not visible on the brain surfaceduring surgery can be visualized by projection of the lesionon the brain surface to plan the approach and operation strat-egy. Future applications of the system could also be for braintumors adjacent to functional areas of the brain which can bevisualized using direct projection of the tumor and functionalMRI results on the brain surface.

The advantage of the presented system compared to theconventional navigation system will be the direct and im-proved visualization of the regions of interest on the pa-tient’s head, skull, or brain. In addition, it is inexpensive andeasy to reproduce. However, further development of thissystem is possible to design a “projection device” for clin-ical applications and it could also be interesting for surgeonsor hospitals that are not able to afford an expensive navigationsystem.

We describe our method, which has been evaluated forprojection of a brain tumor. Furthermore, it could also beinteresting for other surgical areas and procedures like sur-gery of spinal tumors or facial surgery.

The registration of the images has been performed man-ually using anatomical landmarks and fiducial markers asdescribed in the paper. The manual registration was veryaccurate with a mean projection error of 0.3 mm, but furthertechnical advancement can enable an automatic registrationand integration into the standard navigation systems andintegration into the microscope during surgery.

As mentioned, the presented paper describes a novel tech-nique for localization of brain tumors for image-guided neu-rosurgery and first evaluation shows an accurate and quickmethod. The next steps are planned to evaluate the accuracy ofthe method in clinical studies with patients performing image-

Fig. 3 Augmented realityusing image projection of acreated virtual image. a Imageprojection of an MRI-based 3Dmodel of the brain surface(MRIcro) with localization ofthe tumor (red). b Projection ofan MRI-based model of thebrain surface with visualizationof superior sagittal sinus andcortical veins (blue) and braintumor (red)

946 Acta Neurochir (2013) 155:943–947

guided brain surgery with this augmented reality system. Webelieve that this new technique will make it possible to projectdirectly the visualized lesions, such as a brain tumor or brain6metastasis, onto the surface of the head, skull, or brain of thepatients. This would be an important improvement of image-guided neurosurgery.

Conclusions

We designed an augmented reality system for direct projectionof a virtual image onto the head, skull, and brain surface in realtime for image-guided neurosurgery. In this paper, the firstevaluation of the system is presented. Further technical devel-opment of this system can be used for image-guided surgery ofbrain lesions and other surgical fields as well. The presentedmethod is easy to reproduce and inexpensive. After the encour-aging visualization results of this augmented reality system,clinical applications are objects of further investigations.

Conflicts of interest None.

References

1. Azuma R (1997) A survey of augmented reality. TeleoperatorsVirtual Environ 6(4):355–385

2. Blackwell M, Nikou C, DiGioia AM, Kanade T (2000) An imageoverlay system for medical data visualization. Med Image Anal4(1):67–72

3. Iseki H, Masutani Y, Iwahara M, Tanikawa T, Muragaki Y, Taira T,Dohi T, Takakura K (1997) Volumegraph (overlaid three-dimensionalimage-guided navigation). Clinical application of augmented reality inneurosurgery. Stereotact Funct Neurosurg 68(1–4 Pt 1):18–24

4. Kawamata T, Iseki H, Shibasaki T, Hori T (2002) Endoscopic aug-mented reality navigation system for endonasal transsphenoidal sur-gery to treat pituitary tumors: technical note. Neurosurgery50(6):1393–1397

5. Kockro RA, Tsai YT, Ng I, Hwang P, Zhu C, Agusanto K, HongLX, Serra L (2009) Dex-ray: augmented reality neurosurgicalnavigation with a handheld video probe. Neurosurgery 65(4):795–807

6. Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, LiraIH, Tagle P (2007) A novel, inexpensive method of imagecoregistration for applications in image-guided surgery using aug-mented reality. Neurosurgery 60(4 Suppl 2):366–371

7. Mahvash M, König R, Urbach H, von Ortzen J, Meyer B, SchrammJ, Schaller C (2006) FLAIR-/T1-/T2-co-registration for image-guided diagnostic and resective epilepsy surgery. Neurosurgery58(1 Suppl):69–75

8. Mahvash M, König R, Wellmer J, Urbach H, Meyer B, Schaller K(2007) Coregistration of digital photography of the human cortex andcranial magnetic resonance imaging for visualization of subduralelectrodes in epilepsy surgery. Neurosurgery 61(5 Suppl 2):340–344

9. MahvashM, König R, Scheef L, Urbach H, von Oertzen J, Meyer B,Schaller C (2004) Multimodale Neuronavigation in der resektivenEpilepsiechirurgie. Bildverarbeitung für die Medizin. Springer,Berlin Heidelberg New York, pp 224–228

10. Maurer CR, Sauer F, Hu B, Bascle B, Geiger B, Wenzel F, RecchiF, Rohlfing T, Brown CM, Bakos RS, Maciunas RJ, Bani-HashemiA (2001) Augmented reality visualization of brain structures withstereo and kinetic depth cues: system description and initial eval-uation with head phantom. Proc SPIE 4319:445–456

Acta Neurochir (2013) 155:943–947 947