AUTOMATIC DETECTION OF ENDOSCOPE IN INTRAOPERATIVE CT IMAGE:APPLICATION TO AUGMENTED REALITY GUIDANCE IN LAPAROSCOPIC SURGERY

Summary of thesis by S. Bernhardt?†‡ Thesis director: Christophe Doignon†

Thesis co-director: Luc Soler?‡ Project supervisor: Stephane Nicolau‡

? IHU, Institut Hospitalo-Universitaire, Strasbourg, FRANCE†ICube (UMR 7357 CNRS), Universite de Strasbourg, Strasbourg, FRANCE

‡IRCAD, Virtual Surg, Strasbourg, FRANCEsylvain.bernhardt@ihu-strasbourg.eu

1. INTRODUCTION

The advent of minimally invasive surgery has changed thelandscape of laparoscopic interventions. This revolution hasbrought many improvements in terms of clinical outcome, butat the cost of the surgeon’s direct vision of the surgical scene.This problem has been alleviated with the introduction of en-doscopic cameras, which would allow the surgeon to guidethe instruments and actions via a digital display.

This reveals to be a great opportunity to artificially augmentthe rendered scene with additional data relevant to the oper-ation, thus improving the surgeon’s experience. This idea ofAugmented Reality (AR) systems for guidance in minimallyinvasive surgery has been the focus of a lot of research for thepast two decades and yet very few commercial products areavailable, in addition to be limited in terms of robustness andaccuracy.

Usually, AR in laparoscopic surgery is achieved by regis-tering a preoperative high definition 3D model (e.g. segmen-tation from CT or MRI images) with the video display fromthe endoscopic camera. This proves to be a challenge, as thedata to be registered may be very different due to several fac-tors. First, the data often originate from different imagingmodalities and dimensions (3D CT grayscale image vs 2DRGB full HD endoscopic sequence). Second, the patient’sanatomy may significantly differ between the preoperativeand intraoperative acquisitions. The organs often shift dueto a different pose for the patient and the intra-abdominal gasinsufflation which applies pressure on the viscera. Moreover,the surgeon’s actions obviously affect the surgical scene as hemobilizes organs or resects tissues.

For the sake of accuracy, an laparoscopic AR system wouldhave to account for this deformation. Many different ap-proaches have been proposed to solve this issue, but most re-quire either some manual interventions from the surgeon orexpensive and impractical external tracking systems. How-ever, one approach seems promising due to the populariza-tion of intraoperative scanners (e.g. CBCT) in so-called “hy-

brid” operating room (OR). An acquisition from such a sys-tem can serve as an intermediary in the registration processbetween the preoperative scan and the endoscopic images andhelp compensate for the deformation between the preoper-ative and intraoperative states. The non-rigid 3D-3D regis-tration between the preoperative and the intraoperative scansis beyond the scope of this thesis, but the literature providesmany promising approaches to this problem.

2. A NOVEL APPROACH

In light of the above, we propose a new paradigm to auto-matically register the referential frame of the intraoperativemodel with that of the endoscopic camera, without any exter-nal tracking system. By including the distal part of the endo-scope within the intraoperative acquisition field and holding itwith an articulated arm, we are able to estimate the directionof the optical axis and the position of the optical center in thereconstructed volume. This approach allows us to determinedirectly the correspondence between the endoscopic cameraand the intraoperative scanner (in our case, an Artis Zeego bySiemens) and thus to register their data (see Fig. 1).

Obviously, this relation holds only when the camera re-mains static, but this may occur several times during an in-tervention such as a liver segmentectomy. Every time the en-doscope is displaced, static AR can be updated by performinganother intraoperative acquisition. If continuous endoscopicnavigation is required, then we could resort to a classic track-ing technique such as SLAM or an external optical trackingapparatus, but our approach would still provide a good ini-tialization for these, which is critical. Finally, provided that agood transfer function and contrast agents allow an improvedvisualization of critical structures in the volume rendering ofthe intraoperative images, we are then able to directly achieveAR without preoperative imaging.

CO2

Laparoscope

Pneumoperitoneum

Liver

C-arm)field)of)acquisition

Patient'sabdomen

Volume)rendering)ofliver)(+preop.)data)

Virtual)camera

Extraction)oflaparoscope

Intraoperative)volume

Fig. 1. Illustration of our AR approach.

3. EXTRACTION OF ENDOSCOPE ININTRAOPERATIVE CBCT DATA

To accurately augment the endoscopic view with intraopera-tive data, we must create a virtual camera with the same viewupon the region of interest (ROI) as the actual endoscope. Thevirtual view will thus match the endoscopic image content.Therefore, we have to determine all the intrinsic and extrin-sic parameters of the endoscopic camera. The intrinsic pa-rameters are composed of the field of view, the position ofthe optical center and the distortion of the lenses. These aredetermined with a classic camera calibration method using acheckerboard.

The extrinsic parameters consist in the location and orien-tation of the camera in the model space (i.e. the 3D imagespace).Orientation: As one can see in the Fig. 2 (left), the intensitypeaks at the location of the surface of the endoscope. There-fore, knowing that it is tubular, we can retrieve the position ofthe endoscope by fitting a 3D cylindrical model on its surface.This is performed by maximizing the average intensity valuesampled in the 3D CT data along this template (see Fig. 2(right)). By considering the average intensity value along thetemplate as an objective function, matching is then optimized.Position: The position of the endoscope tip is found by de-tecting a large drop in the intensity profile along the deter-mined orientation axis. It is worth noting that the resultingtip location will not actually correspond to the location of thevirtual camera on the axis, notably because it depends on thezoom factor and the focus. Nonetheless, calibration experi-ments have shown that, for standard FOV and focus settingsfor such interventions, the correct location is very close to theactual tip of the endoscope and does not vary significantlyacross experiments.Roll angle: This angle is automatically estimated by using ahigh-end 3-axis accelerometer designed for low accelerationsand included in the camera. It is aligned and soldered on theprinted circuit board (PCB) together with the imaging sensor,so the roll angle measured by the accelerometer is also that of

the camera with respect to the gravity field. The CBCT sys-tem is very precisely calibrated to provide a volume with itsvertical axis aligned with the Earth’s gravity. Thus, the rollangle of the virtual camera with respect to the 3D CT datacan be set to the value given by the accelerometer. Experi-ments showed that the accelerometer provides an accuracy of±0.15◦, which is sufficient for our purpose.

4. REGISTRATION REFINEMENT

So far, we assumed that the optical axis perfectly coincideswith the revolution axis of the endoscope, but this may be vi-olated depending on the model used. This difference, thoughvery small at the scale of the device, may result in several tensof pixels of registration error in the augmentation.

4.1. Dedicated calibration

We designed a calibration method to detect such a misalign-ment and compensate it in the image plane. As seen in Fig.3, by making a square tube rotate around the distal end of theendoscope, it is possible to determine the intersection of theaxis of revolution with the image plane at the point C’. Sincewe position our virtual camera centered on this axis, the cen-ter of the virtual image ought to coincide with the center ofthe real endoscopic image at the point C. This is enforced bya trivial translation between C and C’ in the image plane.

This calibration is simple and can be performed intraop-eratively within 30s, thus not interfering too much with thesurgical workflow.Results:We tested our AR method on a radio-opaque checkerboardboth visible from the scanner and the endoscope. Across tensof experiments, the registration error was in average smallerthan 15 pixels in a 1080p image which is equivalent to lessthan a millimeter in the scene at the nominal distance (around7 cm). Other statistical analyses were carried out in order toconfirm the robustness of our method and are detailed in thethesis.

Endoscope

(>6000)

Contrastyagent

(<4500)

CO2

(-1024)Liver

(<200)

Bone

(<1500)

C

R

φ

θ

xy

z

Intensity-1024 5000 10000 15000

Fig. 2. Left: The image on the left is a slice of the CBCT image, going through both the endoscope and a pig’s anatomy. Theintensity values are displayed for different regions of the scene, showing that the endoscope voxels exhibit very high contrastwith any other part of the volume. Right: This image shows the cylindrical template over the same slice. The sampling points(orange) are positioned with respect to a point C on the axis of revolution, the Euler angles φ and θ and the radius R.

Δ

Δ

C'

C'

C

Fig. 3. Left: A square-profiled tube rests on the distal part of the endoscope. The line passing by the top and bottom corners(green) naturally passes by the sought axis of revolution of the endoscope ∆. Right: As the square tube rotates around theendoscope tip, all green lines are easily detected and intersect at ∆, which is C’ in the image (red).

We also performed our method offline on several in vivodata from interventions on pigs. It confirmed the aforemen-tioned sub-millimetric accuracy and showed the clinical po-tential of our approach.

4.2. Shading-based refinement

While the previous calibration method is simple, it is notfully automatic and requires an operator to rotate the squaretube. Moreover, most endoscopes do not have integrated ac-celerometer and then the roll angle remains unknown. Giventhese two drawbacks, we also designed a registration refine-

ment method that automatically aligns the virtual view and thereal one based on image analysis only. Essentially, this con-sists in determining the 2D rotation and translation betweenthe two images, thus discarding the need for the square tubecalibration and the accelerometer.

The following method results from a collaboration with Dr.Adrien Bartoli from ALCoV-ISIT at Universite d’Auvergne inClermont-Ferrand, France.We propose to obtain these by optimizing a dissimilarity met-ric between the laparoscopic image and the view from thevirtual camera upon the extracted surface of intra-abdominalcavity, using local shading constraints. Since the cavity is in-

sufflated with carbon dioxide, it presents a good contrast withthe surrounding tissues and therefore extracting its surfacefrom the intraoperative 3D data is trivial, using for instancemarching cubes.

We first design a simple lighting model for the endoscope,assuming a point source at its tip. This model is valid for ar-eas that are not extremely lit or unlit and for which the albedois approximately constant. Therefore, while discarding spec-ularities or dark areas, we also divide the image into a set ofhomogeneous patches using the watershed algorithm. Most ofthe patches are large enough to contain shading informationand thus constrain the dissimilarity measurement. By gather-ing the constraints from each patch into a cost function, weare able to optimize the similarity between the laparoscopicimage and the view of its extracted surface, which is equiv-alent to determining the sought translation and rotation (seeFig. 4).Results:From offline in vivo experiments, we performed manual mea-surements of the alignment error by pointing visual cues suchas edges or corners in both images. Our method presented anaverage registration error of 11.3 ± 4.7 pixels in the image,which corresponds to less than a millimeter in the scene atnominal distance. Thus, the complete relationship betweenthe laparoscopic image and the intraoperative 3D data is de-termined, without additional apparatus or calibration. How-ever, typical optimization computation times range from 15 to30s on a standard PC. Added to the initial step of endoscopeextraction, the complete augmentation process takes between25 to 55s. We plan to optimize the whole process and make itfaster for a better clinical integration.

5. CONCLUSION

We present a novel approach to AR in laparoscopic surgery.Based only on standard hybrid OR equipment, our methodcan provide the static relationship between the endoscope andan intraoperative 3D scan, up to a rotation and a translationin the image plane. We propose two ways to complete thispartial registration. One is a simple and fast dedicated cali-bration process that requires only a square tube. The other ispurely based on image analysis and uses an original shadingformulation with piece-wise constant albedo, best suited forlaparoscopic scenes.

A preliminary version of this method has first been reportedin [1] and then in [2]. A more advanced and detailed versionof our work, including the square tube calibration, is to bepublished as [3]. Finally, the registration refinement methodbased on shading is currently under review to be published as[4].

Fig. 4. The laparoscopic image (top) is registered with thevirtual view of the surface extracted from the intraoperative3D data (middle top). Mosaicing of the two shows the align-ment before the proposed optimization (middle bottom) andafter (bottom).

Publications[1] S. Bernhardt, S. Nicolau, V. Agnus, L. Soler, C. Doignon, and J. Marescaux. Automatic detection of endoscope in intraop-

erative CT image: Application to ar guidance in laparoscopic surgery. In Proceedings of ISBI, pages 563–567. IEEE, 2014.Oral presentation.

[2] S. Bernhardt, S. Nicolau, V. Agnus, L. Soler, C. Doignon, and J. Marescaux. Automatic detection of endoscope in intra-operative CT image: Application to ar guidance in laparoscopic surgery. In Proceedings of Surgetica, pages 95–97, 2014.Oral presentation.

[3] S. Bernhardt, S. Nicolau, V. Agnus, L. Soler, C. Doignon, and J. Marescaux. Automatic localization of endoscope inintraoperative ct image: A simple approach to augmented reality guidance in laparoscopic surgery. Medical Image Analysis,2015. Accepted, to be published.

[4] S. Bernhardt, S. Nicolau, A. Bartoli, V. Agnus, L. Soler, and C. Doignon. Using shading to register an intraoperative ctscan to a laparoscopic image. In Proceedings of CARE workshop, MICCAI, 2015. Under review.

Publications avant la thèse de doctorat

S. Bernhardt, J. Boutet, L. Herve, D. Vray, Bimodality US/optical imaging: registration and 2D/3D

visualization, Proceedings of IEEE International Ultrasonics Symposium, October 2010.

S. Bernhardt, J. Abi-Nahed, R. Abugharbieh, Robust Dense Endoscopic Stereo Reconstruction for

Minimally Invasive Surgery, Proceedings of the MICCAI workshop on Medical Computer Vision,

October 2012.