Integration of new features for telerobotic surgery into the …downloads.hindawi.com/journals/abb/2011/635951.pdf · 2019. 7. 31. · Applied Bionics and Biomechanics 8 (2011) 253–265

Applied Bionics and Biomechanics 8 (2011) 253–265DOI 10.3233/ABB-2011-0037IOS Press

253

Integration of new features for teleroboticsurgery into the MiroSurge system

Rainer Konietschkea,∗, Davide Zerbatob, Rogério Richac, Andreas Tobergtea, Philippe Poignetc,Florian A. Fröhlicha, Debora Botturib, Paolo Fiorinib and Gerd HirzingeraaInstitute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, GermanybDepartment of Computer Science, University of Verona, Verona, ItalycRobotics Department, LIRMM CNRS/Université Montpellier II, France

Abstract. Minimally invasive robotic surgery has gained wide acceptance recently. Computer-aided features to assist the surgeonduring these interventions may help to develop safer, faster, and more accurate procedures. Especially physiological motioncompensation of the beating heart and online soft tissue modelling are promising features that were developed recently. Thispaper presents the integration of these new features into the minimally invasive robotic surgery platform MiroSurge. A central aimof this research platform is to enable evaluation and comparison of new functionalities for minimally invasive robotic surgery.The system structure of MiroSurge is presented as well as the interfaces for the new functionalities. Some details about themodules for motion tracking and for soft tissue simulation are given. Results are shown with an experimental setup that includesa heart motion simulator and dedicated silicone organ models. Both features are integrated seamlessly and work reliably in thechosen setup. The MiroSurge platform thus shows the potential to provide valuable results in evaluating new functionalities forminimally invasive robotic surgery.

Keywords: Medical robotics, minimally invasive robotic surgery, soft tissue simulation, heart motion tracking and compensation

1. Introduction

Robotic assistance in minimally invasive surgery hasbeen a promising approach for many years. But onlyrecently it achieved wide acceptance in e.g. radicalprostatectomy due to advantages like less invasive-ness and a shorter hospital stay [35]. The only systemfor these interventions which is commercially avail-able today is the daVinci® product line of IntuitiveSurgical®. It is structured into a patient-side slavecart and a master console to teleoperate the slave.The robotic slave side offers up to four arms, dex-

∗Corresponding author: Rainer Konietschke, Institute ofRobotics and Mechatronics, German Aerospace Center (DLR),Münchnerstrasse 20, 82234 Wessling, Germany. E-mail: [email protected].

trous instruments and stereo imaging. On the masterside, the surgeon gets a stereo vision of the scene andcommands the slave side via hand controllers, buttonsand foot pedals. The telemanipulation is enhanced byadditional features like motion scaling and tremor fil-tering. Various research platforms were developed upto now that aim at extending these functionalities indifferent ways, see e.g. [45, 17] This paper presentsas a result of the European project AccuRobAs theintegration of two new features that are widely con-sidered beneficial. Motion compensation controls therobotic system in a way that the surgeon may operateon a virtually still standing region of interest (ROI) onan organ. Online integration of soft tissue simulationmakes a virtual preview of certain operation steps pos-sible. Both features did not have strong relevance in

1176-2322/11/$27.50 © 2011 – IOS Press and the authors. All rights reserved

mailto:Rainer.{penalty -@M }[email protected]

254 R. Konietschke et al. / Integration of new features for telerobotic surgery into the MiroSurge system

industrial robotics until now, where usually the work-piece was in rest, or had a well known motion, andcould be described by simple and known material prop-erties. However, the relevance of these features in thecontext of surgical robotics is evident. Recently, fea-tures that enable to work in unstructured workcells alsogain relevance in industrial robotics e.g. when cooper-ating with the human. In surgical teleoperation, thebenefits that come from the physical simulations aremanifold [5] The virtual environment can be used dur-ing a pre-operative planning phase but it is also usefulto integrate supplemental information during the actualintervention. In particular augmented reality obtainedwith the superimposition of the simulated scene overthe real camera view can help the surgeon in navigatingthe operating area by highlighting important or hiddenstructures (e.g. vessels, tumors). With virtual fixturescertain areas can be prohibited. In addition, the forcescomputed by the physical simulation can be used towarn the user about wrong or dangerous gestures orrendered to the user to increase perception of the realenvironment [23].

This paper describes the integration of the featuresmotion tracking of the beating heart and online softtissue modelling into the MiroSurge research platformshown in Fig. 1. The MiroSurge platform, developed atthe German Aerospace Center (DLR), enables researchon minimally invasive robotic surgery. It follows amodular design both on slave and master side. Onslave side (see Fig. 1, left), MIRO robots [16] realizethe extracorporal motion for positioning two instru-ments and a stereo endoscope. Inside the body, theinstruments are equipped with a three degree of free-dom (DoF) actuation to allow for full dexterity andan additional function such as gripping. The tip of theinstrument furthermore integrates a seven DoF sensorto measure contact forces and torques as well as grip-

ping forces. The forces are fed back to the master side(see Fig. 1, right) either for visual augmentation orfor haptic rendering with a device such as the showncommercially available omega.7 input devices [11].

Various features to assist the surgeon and to increasethe operation quality are currently researched. Thesefeatures either aim to reduce the system’s complex-ity (e.g. setup planning), increase the patient safety(e.g. virtual walls), enhance the surgeon accuracy (e.g.intelligent motion scaling), or provide additional infor-mation (e.g. visual augmentation, or force feedback).Research on the MiroSurge platform intends to includeall these aspects. Namely, an optimization and plan-ning software assists the surgeon during system setupand patient registration [22], and the surgeon receivesforce feedback during the operation, either visuallyaugmented or directly rendered to the haptic inputdevices [16]. The features motion tracking of the beat-ing heart and online soft tissue modelling have thepotential to further enhance the MiroSurge system andare introduced in the next paragraphs.

Motion tracking. Although current surgical plat-forms have considerably improved the ergonomics andmobility issues related to the minimally invasive proce-dure, physiological motion still needs to be manuallycompensated by the surgeon. For tackling this issue,an active motion compensation can be employed forenabling surgeons to operate directly on moving organswith high accuracy and low cognitive burden.

Unlike passive motion compensation solutions suchas mechanical stabilizers, active motion compensationsystems are based on the online estimation of phys-iological motions. Solutions vary from mechanicalsystems such as the moving support proposed by Tre-jos et al. [48] to augmented reality systems [42], wherea virtually stable view is rendered for the surgeon. Allstudies report a significant increase in the precision

Fig. 1. The MiroSurge System: Slave side (left) and master side (right).

R. Konietschke et al. / Integration of new features for telerobotic surgery into the MiroSurge system 255

and repeatability of the surgical gesture, attesting thepractical value of a motion compensation system inrobotized surgery.

A fundamental requirement in a motion compensa-tion system is the accurate estimation of physiologicalmotions. In the system that integrates the MiroSurgeplatform, the visual feedback provided by the endo-scope is used for motion tracking. From a practicalpoint of view, passive vision-based techniques are pre-ferred for avoiding contact and the introduction ofadditional sensors in the limited surgical workspace.To the authors’ knowledge, this is the first vision-based3D motion compensation system to incorporate a sur-gical platform that does not require the insertion ofartificial markers for facilitating the visual trackingtask.

In the context of motion compensation, cardiacsurgery represents the most challenging scenario insidethe human body due to the complex heart dynamicsand high precision requirements. In literature, severalvisual tracking approaches for estimating the beatingheart motion can be found. Techniques can be dividedinto two classes: feature-based or region-based track-ing.

Feature-based methods [44] can be used to tracksalient features such as blood vessels or other distin-guishable structures for retrieving the heart motion.Although various studies suggest the feasibility ofmotion estimation using this class of methods, trackinghas limited performance against large tissue defor-mations and appearance changes. Furthermore, it isstrongly dependent on the availability of stable fea-tures. Its main advantage is the low computationalrequirements, which enables real-time performance.

On the other hand, in region-based methods suchas [36] tracing is formulated as an iterative regis-tration problem. Given the assumption that the heartsurface is smooth, continuous and sufficiently tex-tured, a parametric function can be used to describe theheart surface deformation. Region-based methods firstappeared as methods for estimating depth in the intra-operative field rather than 3D motion tracking. Lau et al[24] proposed a B-spline parametric model to retrievedepth from the disparity between stereo images of abeating heart. Another approach for dense 3D depthrecovery was proposed by Stoyanov et al. [43] wherea piecewise bilinear mapping is applied for modelingsoft-tissue deformation. A challenge when using thesemethods is the handling of specular highlights whichrequires additional preprocessing [14].

The issues raised above concern the complicatedproblem of finding a suitable deformation model forthe heart surface. A possible solution can be physi-cally inspired models such as the membrane modelproposed by Bader et al. [2]. However, convincingexperimental results are still needed to demonstrate thevalidity of such models. In addition, results of shape-from-shading (SfS) techniques for reconstructing theheart surface have also been reported in Lo et al. [30,31]. Compared to the computation stereo approachespresented earlier which perform well in regions withdistinctive features and sufficient texture, shape-from-shading tends to perform well in regions with uniformalbedo, little texture and smooth local curvature. SinceSfS and computational stereo techniques are comple-mentary, a probabilistic fusion framework based onMarkov Random Fields (MRF) was proposed. Theintegration of human perception through gazecontin-gence in a similar fusion framework was recentlyproposed in Visentini-Scarzanella et al. [49]. Never-theless, the uniform albedo and Lambertian surfacerequirement for SfS to provide reliable results are notoften met in MIS images. Furthermore, the computa-tional requirements are still an issue when real-timeperformance is envisaged.

From the methods cited above, a region-basedmethod based on the Thin-Plate Spline (TPS)deformable model was chosen for the motion compen-sation system. The most interesting feature of the TPSmodel is the flexibility in parameterizing deformations,which allows us to overcome problems concerning thelack of visual information in regions with poor tex-ture. Tracking is formulated as an iterative registrationproblem using a gradient-based minimization tech-nique called ESM [8] extended to the stereo scenariofor tracking in 3D. Further implementation details aregiven in Section 2.2.

Online soft tissue modelling. The simulation ofdeformations represents an important aspect in manyfields. From structural analysis to 3D modeling theneed for realistic or fast simulation of deformationslead to the development of many different approaches.In the last few years, increased computational powerand advances in research lead to the introduction of softor deformable models into virtual interactive environ-ments. In the medical field this translates into morerealism for surgeon training or intervention planning.A key aspect of interactive deformable models isthe computational complexity, and thus the optimizedimplementation of the simulation. For this reason many


different models were developed and some parallelimplementations were proposed.

Deformable models can be roughly divided intothree main classes: finite element models (FEM),meshless models and mass spring models (MSM). Twomain issues arise from simulation of deformable mod-els: physics realism requires small temporal steps inthe simulation, in addition haptic feedback should beupdated every 1 msec to be smooth and realistic, thusinteractive simulations with haptic feedback imposevery tight constraints on the computational time andrequire optimized implementations.

FEMs can give the most accurate results in sim-ulations of deformable tissues, they are based on acontinuum representation of the tissue and on materialconstitutive equations [12]. This makes them realisticand easy to calibrate. Their use is widespread in offlineengineering and surgery simulations where accuracyis achieved at high computational cost [20]. LinearFEM are used in online simulation but their realismis reduced to small deformations and rotations [6].

Meshless models (or point based methods) share thetheoretical background with FEM and have been usedin simulation of different materials [34, 19] and also insurgical simulations [27, 29]. The main advantage ofpoint based models is their ability to handle cuts, andmore generally, changes in topology. Along with theirhigh computational cost, one of the main drawbacksof point based models is the difficulty in handling sur-face and interactions because modelling of physics andsurface is decoupled.

A different approach leads to MSMs [37] which havebeen widely used in surgical simulations [1, 33]. InMSMs the body is discretized in a cloud of mass pointslinked by springs and dampers. A displacement in amass position leads to an extension or a compressionof one or several springs. This produces internal forcesthat are integrated temporally and leads to a differ-ent configuration of the body. The main disadvantageusing MSM is the difficulty in their calibration, sincetheir physical background is not as evident as FEM andthe definition of spring and damper coefficients is notstraightforward. Their reduced computational com-plexity makes them suitable for interactive, physicallybased, simulation of very complex models, moreoverMSMs have been effectively ported on graphics pro-cessing units (GPUs), fully exploiting their parallelismand memory management. For these reasons the workpresented here focuses on the use of MSM in conjunc-tion with a GPU.

Many works have been proposed about the use of theGPU to speed up computations of deformable environ-ments simulation [18, 50]. An implementation of MSMon GPU is presented in [13]: masses are allocated ona 2D texture and springs on a group of 2D textureswhere each element stores a spring connected to a massstored on the same position. In [1] a method for phys-ical simulation that relies on GPU and on MSMs tomimic in real time the physics of human organs is pre-sented. The work proposes an innovative node orderingmethod and optimized spring forces computation thatallow to obtain online simulation.

The remainder of this paper is organized as fol-lows: In Sec. 2, the developed algorithms are presented,namely the integration into the MiroSurge system (Sec.2.1.), the motion tracking (Sec. 2.2.), and the online softtissue modelling (Sec. 2.3.). Preliminary experimentalresults are shown in Secs. 3, 4 concludes the paper.

2. Methods

The next Subsection describes how the new fea-tures are embedded into the Mirosurge control scheme.Then, the modules Tracking of the beating heart andSoft Tissue Modelling are further detailled.

2.1. Integration into MiroSurge

The features presented in this paper need to beintegrated into the basic scheme of the so far imple-mented force feedback telemanipulation control shownin Fig. 2. The bilateral telemanipulation implements aposition-force architecture with motion scaling. Carte-sian velocities measured at the master are scaled downby a factor of 2–4 and integrated on the slave’s initialpose. The motions from the haptic device are trans-formed in a way that an intutive hand-eye-coordinationis given for the user in the sense that moving of the hap-tic device in one direction corresponds to the motionthe user sees in the stereo image. The measured inter-action forces returned from the slave are amplified andcommanded to the master. The gain factor dependson the motion scaling [47]. An analytic inverse kine-matics for the MIRO and the instrument calculatesthe joint angles such that the instrument crosses theentry point and reaches the commanded slave pose.The analytic solution is enhanced by a numeric opti-mization for the 1 DoF nullspace to avoid collision ofthe robotic arms, singularities and joint limits [21]. The


Fig. 2. Basic scheme of force feedback telemanipulation for the MiroSurge platform.

desired joint angles are sent to the slave’s controllerwhich takes care of fast motion tracking. For the con-trol of the MIRO robots, a full-state feedback controlfor flexible and coupled joints is implemented [25].The controlled states are the motor positions and jointtorques which are both directly measured, as well asthe derivatives of both measurements. An additionaldisturbance observer uses the joint torque measure-ments to reduce friction in the motor and the gearboxes [26].

Motion compensation. To integrate motion compen-sation, the target motion is captured by a stereo cameraand fed to the heart tracking module as shown in Fig. 3.The heart tracking module has to generate as output theCartesian velocity of an ROI vROI, written in the coor-

dinate system (COS) of the camera. This velocity isthen transformed into the COS of the instrument toolcenter point (TCP). Depending on whether the motioncompensation is turned on (H = 1) or off (H = 0),the tracked velocity is added to the Cartesian velocityvUSER coming from the input device.

Online soft tissue simulation. The user has the optionto either interact with the real organ or with the sim-ulation of the organ, see switch S in Fig. 3. In bothcases force feedback is available. In case the userinteracts with the real scenario (S = 0), the simula-tion is running in parallel and provides monitoring.This way, large discrepancies between the measuredreal interaction forces and the forces calculated in thesimulation may be used as indicator for sensor mal-

Fig. 3. Integration of the additional features motion compensation and online soft tissue simulation into the Mirosurge control scheme.


function or bad registration. If the user is interactingwith the simulation (S = 1), the robotic system is notmoving. When switching back from the simulation toreal interaction, the integrator has to be reset to theactual instrument TCP. For simulation, the simulatedslave is idealized by simple forward kinematics, thusassuming that the virtual robot instantaneously reactsto desired motions. In future work, this can be replacedby a more complex physical model of the roboticsystem.

2.2. Tracking the beating heart

Due to the complex nature of the heart surface, asuitable choice for a deformation model is complicatedas shown in Sec. 1. The parametric models currentlyused to describe the heart surface display problemswith specular reflections and textureless (homoge-neous) regions. For instance, the B-spline mappingin [24] does not employ any surface regularization,which may lead to erroneous shape estimates in regionswithout sufficient texture. Similarly, the bilinear map-ping in [43] needs a very large number of controlpoints to sufficiently approximate the heart surfaceshape.

For such reasons, a type of Radial Basis func-tion called the Thin-Plate Splines (TPS) for modelingthe heart surface deformations is chosen. The TPSwas successfully applied for modeling non-rigid tis-sue deformation [4, 3, 28]. One of the most interestingfeatures of the TPS mapping is the flexibility in placingits control points. This represents the key feature thatallows to overcome the difficulties of previous meth-ods concerning the lack of visual information whentracking regions with poor texture.

Based on extensions to the original TPS affine for-mulation [32, 4] a region-based method for tracking theheart surface deformations with a novel parameteriza-tion for 3D tracking in the stereo scenario using the TPSwas developed [38]. This parameterization offers anefficient formulation of the tracking problem, withoutrequiring disparity search or other intermediary stepsthat may reduce the tracking accuracy.

Initially, a smooth and continuous region of inter-est on the heart surface is selected as the referenceimage T. Next, to model the heart surface deforma-tion, a Thin-Plate Spline parametric model is used. TheTPS is a Radial Basis Function characterized by thebasis function U(s) = s2log(s2), (n + 3) parameters(w1, . . . , wn, r1, r2, r3)T and a set of n control points

ci = (x̂i, ŷi), such that a spatial mapping of pixels x �→ f(x) is calculated as:

f (x) = r1 + r2x + r3y +n∑

i = 1w iU (||ci − x||)

where ||·|| denotes the Euclidean norm. The controlpoints define the degrees of freedom of the warpingmodel (more or less control points can be used toaccount for the local heart surface deformation).

In the monocular case, two TPS functions f x andf y sharing their control points define a mapping of thepixel positions of the selected reference image ontothe current image of the operating field. For trackingin 3D, we seek to align the same reference image onboth stereo images simultaneously. Consequently, aTPS warping w(x, h, C) of a pixel x can be definedas function of a vector h of Cartesian coordinates ofpoints in space that map to the control points on eachstereo image and the respective camera matrix C (formore details, see [38]).

Consequently, the 3D tracking problem consists inestimating the optimal warping parameter vector h thatminimizes the alignment error � between the referenceimage T and both left and right images of the stereopair Il and Ir simultaneously:

minh

� =∑

x∈A

[[Il(w(x, h, Cl)) − T(x)]2

+[Ir(w(x, h, Cr)) − T(x)]2]

where A is the set of the template coordinates andIl(w(x,h,C)) and Ir(w(x,h,C)) are the current left orright images transformed by the warping functionw(x,h,C), respectively.

For solving the minimization problem above, we usethe efficient second-order minimization (ESM) algo-rithm proposed by [7]. The ESM is applied because itdisplays a faster convergence rate and larger conver-gence basin than traditional optimization techniquessuch as Gauss-Newton or Levenberg-Marquardt. Inaddition, for increasing the robustness towards illu-mination variations the illumination compensationmethod proposed by Silveira et al. [8] is incorporatedto the tracking framework. Tracking results obtainedwith the proposed technique on an in vivo beating heartare illustrated in figure 4. A schematic overview of thetracking algorithm is given in Fig. 5.

Implementation on a Graphics Processor Unit(GPU). The visual tracking algorithm must extract


z (m

m)

z (m

m)

x (mm) x (mm)

y (mm

)

y (mm

)

Fig. 4. Endoscopic stereo images of a porcine beating heart, acquired at high speed (83 Hz) with a very small baseline [39]. Notice the difficultiesinvolved in acquiring at high speeds due to insufficient lighting -For different instants of the heart cycle, the target region tracked on the leftand right images of the stereo pair and the corresponding TPS surface approximation are illustrated. The reconstruction uses the left camera asworld coordinate frame. The tracking algorithm is robust to illumination variations and specular reflections.

Fig. 5. A schematic overview of the tracking algorithm.

the heart motion on-line for an accurate synchro-nization with the robotic tools. This implies thattracking must run at high frame-rates. The firstefforts towards a fast implementation of the track-ing algorithm were far below the desired trackingframe-rate (≈ 5 Hz in an implementation using IntelPerformance Primitives -IPP). Due to the largercomputational requirements of the application, the

computational power of recently released GPUs isexplored.

The market for real-time high-definition 3D graph-ics has given birth to high-parallel, multi-coreprocessors with great computation power. Since GPUsare specially well-suited to address problems that canbe expressed as data-parallel computations, an imple-mentation of the tracking algorithm was developed inorder to take advantage of the hardware efficient pro-cessing. For the experiments, a NVidia GTX280 (SantaClara, EUA) graphics card programmed using CUDA(a programming extension to C) was used.

The computation time per iteration may varydepending on the size of the reference image, thenumber of spatial and illumination parameters. Forinstance, considering a 128 × 128 pixel region of inter-est with 5 control points the computation time periteration required for tracking is of ≈2.25 ms. Assum-ing a transfer delay of around 0.2 ms from the camerasto the graphics card memory, tracking speeds over50 Hz can be achieved (a × 10 speed improvementin comparison with the CPU implementation). It isnoticeable that the number of iterations required for thetracking algorithm to converge depends on the inter-


frame motion, which is directly related to the imageacquisition speed. If the acquisition speed is increased,the heart displacement between frames is reduced andless iterations are necessary for tracking to converge.Therefore, the computational requirement per framedecreases with an increase of the acquisition speed.

2.3. Soft tissue modelling

The workflow followed to provide online simulationof human organs starts from the CT acquisition of thepatient. In the scope of this paper, a silicone model ofa liver with stiffer inclusions is used (see Sec. 3). Thesubsequent steps are CT image segmentation, MSMgeneration and, finally, the actual simulation. Duringthe whole process a rigid structure firmly attached tothe model provides a common frame of reference forthe different phases.

Following the semi-automatic procedure describedin [10], the liver and the inclusions is segmented.To obtain the segmentation three steps are applied:liver segmentation using a watershed based algorithm,optimal inclusion threshold identification by entropyminimization and segmentation refinement with holefilling and surface smoothing. The overall procedurerequires some manual tuning of parameters due to thedifferences in Hounsfield values between real tissuesand synthetic phantom.

The different volumes (liver and inclusions) identi-fied by the segmentation phase are discretized into asingle tetrahedral mesh. A label is associated to eachtetrahedra to keep track of the constitutive material(i.e. liver parenchyma, tumors, ...). From this mesh theMSM structure is simply obtained by placing a masspoint in each vertex of the mesh and a spring-dampersystem in correspondence of each mesh edge. Tetra-hedra labels and original CT data are used during thesubsequent calibration phase. In the specific case of thesynthethic model the imporant parameters of the dif-ferent material needed for its realization are identified:density and stiffness. These parameters are used with acalibration method based on the work proposed in [51]to associate a mass value to each model vertex and elas-tic and damping coefficients to each model spring. Keyfeatures of the obtained model can be found in Table 1,and the simulated organ is shown in Fig. 6.

In order to achieve a registration of the virtual organwith the real one, a surface scan of the organ is per-formed with the 3D-Modeller (3DMo). The 3DMo isa handheld device for capturing the surface of objects.

Table 1Key features of the synthetic liver model

Complete Mesh Liver Inclusion

Masses Springs Density Elasticity Density Elasticity16398 90592 1326 Kg/m3 UkPa 1326 Kg f/m5 31 kPa

Fig. 6. The synthetic liver model being manipulated by virtualrobots.

The generated mesh is then matched with the vir-tual organ surface automatically [22]. The method wasdeveloped for open surgery. In a minimally invasivesurgery the organs are not accessible to the 3DMo dueto the size of the device, therefore new algorithms thatallow to replace the 3DMo by a stereo camera systemare currently adopted.

The last phase of this process is the online simulationof the obtained deformable model. The simulation runsalong with the reality and mimics the effects of surgeonaction on the real models, the realism of the simulationallows the user to seamlessy switch between real andvirtual environment.

To exploit GPU computational power data isencoded in a format that allows parallel processingand that follows the limitations imposed by GPU mem-ory management. The presented simulator is based andextends the work proposed in [1] to which the readeris referred. Each simulation step is composed by threephases: collision detection and solution, internal forcecomputation, temporal integration of model configu-ration. All those phases are carried out by the GPU,the only data exchanged with CPU are the virtual toolposition and the resulting interaction force.

The use of the GPU allows the physical simulationto run at more than 1 kHz and the graphics of the sceneto be rendered at 30 f ps. This ensures the realism ofboth haptic and graphic feedback.


Fig. 7. Heart motion simulator (left) and plot of the desired motion of the heart simulator and the tracked motion (dashed), (right).

3. Preliminary experimental validations

A custom made silicone model of the heart is usedas target for tracking. The heart model is mounted toa parallel robot that allows for translational motions.As target motion, a dual sinus in the image planeof the endoscope is assumed. It models the heartcontractions with frequency fheartbeat = 1.25 Hz, ampli-tude aheartbeat = 1 mm as well as the breathing withfbreathing = 0.2 Hz and amplitude abreathing = 1.4 mm.Harmonics of these base frequencies are not included,and there is a zero offset between the two sinusses.Despite these simplifications, the resulting motionresembles a real heart motion as shown e.g. in [46,39]. Figure 7 (left) shows the heart motion simulator.The heart motion is captured by a stereo endoscopewith 25 Hz.

To evaluate the online soft tissue modelling, a sili-cone model of a liver with included tumors was built,see Fig. 8. The physical parameters for healthy tissueand for tumors are taken from literature. Both organsare placed according to the human anatomy beneatha transparent replication of a human torso, see Fig. 9(right).

Motion compensation. The desired motion of theheart simulator and the tracked motion of the ROIare shown in Fig. 7 (right). The motion tracking reli-ably computes the velocities of the ROI in less than120 ms. The motion is then fed to the robot controlto achieve the motion compensation behaviour of therobot. Since no motion prediction is included, a slightdelay between the desired and tracked motion can beseen in Fig. 7 (right).

Fig. 8. Silicone liver model.

The delay between the image acquisition and visualtracking step to the robot actuation can be com-pensated by exploiting the quasi-periodicity of thebeating heart motion for predicting its future motion.Although no motion prediction is incorporated tothe current system, studies show that low predictionerrors can be achieved in small prediction hori-zons (see [40] such as the measured compensationdelay (120 ms).

Online soft tissue modelling. The described setupproved that seamless switching between the real andthe simulated environment is possible and useful. Aqualitative comparison between contact forces showedthat tumor localization was possible both in the realand simulated environment. The findings are consis-tent with results from manual palpation. Figures 9 and10 show the integrated setup. The integration of theinteractive, physically based simulator in the architec-


Fig. 9. Integration of the additional feature online soft tissue simulation into the MiroSurge platform. On the left the screen arrangement isshown: The front screen provides the actual image from the real stereo endoscope, while the screen on top provides a rendered scene. The slaveside of the MiroSurge system during a palpation task is shown on the right.

ture provides some benefits to the overall procedure.During our tests the simulator received the input fromthe master controllers and from the camera controller.Physics simulation was updated at a frequency of morethan 1200 Hz with a synthetic liver model composedby more than 16,000 mass points and about 90000spring-damper systems. Subjective evaluation of hap-tic feedback provided by surgeons proved that thedeformable model response is comparable with the realone. Objective validation of simulated data in realis-tic environment will require more advanced techniquesand should be performed on real biologic tissues.

During the described test simulator efficientlyprovides three enhancements to the interventional sce-nario. It allows the surgeon to control a virtual camera

in free motion inside the scene, to obtain a clearer viewof the interventional area and to better understand therelative position of tools and organ of interest. Theaddition of transparencies to the rendered scene allowsthe user to see the inner structures of the organs. In thespecific case (see Fig. 10, right), it is possible to dis-play the inclusions of the synthetic liver, thus guidingthe user during the intervention. Last, the developedarchitecture allows the user to switch between the realscenario and the virtual one, to try some actions insimulation before performing them in reality. Resultsprove that the simulation can run along with the reality,thus satisfying required real time constraints, more-over the simulation correctly mimics the reality in bothgraphic and haptic rendering.

Fig. 10. Rendered scene while monitoring the motion of the robotic tool (left) and transparent rendering to show the inclusions inside the liver(right).


4. Conclusion

New modules for motion compensation for beatingheart surgery and online soft tissue modelling weresuccessfully integrated into the Mirosurge platform.

Motion compensation is an important feature toincrease surgeon accuracy when operating on mov-ing organs. To the authors’ knowledge, this is thefirst vision-based 3D motion compensation system toincorporate a surgical platform. This paper shows thefeasibility of its integration into a complex telemanip-ulation system in 6 DoF with dedicated kinematics forthe entry point into the patient. To further enhance thequality of the motion compensation, a higher imageresolution of the endoscope would be beneficial forcapturing the details of fine heart structures. Anotherimportant issue is the processing delay which includesthe image acquisition by the endoscope, tracking,control and the actuation stages. If this delay is notsufficiently low, critical errors can be induced in themotion compensation system. Even though palliativesolutions such as motion prediction can compensatefor this delay in most of the cases, the compensationerrors due to latency should be kept within safe levels(under 200 �m) required by cardiac surgery [9]. In fact,recent studies show that low prediction errors can beachieved in small prediction horizons (see [40]) such asthe measured compensation delay (120 ms). The eval-uation of motion prediction methods (as developed e.gin. [36]) with the MiroSurge platform is future work.These methods could also be used to bridge situationswhen the view of the camera to the ROI is occluded bythe instruments. In addition, once the instruments arein contact with the organ, an approach using the mea-surement of contact forces as well as vision for motioncompensation may be advantageous. To provide theuser with a view on the virtually still standing organ, thevelocities generated from the motion tracking also needto be fed into the visualization at the master console.

The accurate estimation of the deformations of inter-nal organs as provided by the tracking algorithm is alsoa key requirement for merging pre-operative and intra-operative data. Patient-specific information such as theinternal organ structures can be used for increasedsurgical safety and surgical guidance once preciseinformation about the online poses of the organ isavailable.

Online soft tissue modeling in the MiroSurge sce-nario has shown potential to assist the user duringoperation. The innovations introduced by our approach

are manifold: the physical modeling of the sceneprovides, in fact, realistic reconstruction of the inter-ventional area that allows to assist the surgeons ineffective ways. In particular the surgeon can freelynavigate inside the scene, thus overcoming endo-scope motion constraints. The surgeon can also exploitadvanced rendering of the scene, that provides theview of hidden structures or inclusions such as tumors,vessels or ducts. The last advantage provided by theproposed method is the ability to switch between thereal environment and the virtual one. This allows theuser to explore the area or to try some parts of the inter-vention without risk for the patient, thus providing abetter understanding of the ongoing surgery.

A crucial challenge remains to achieve online regis-tration of the organ pose and of its physical properties,especially in case the organ is changed during operatione.g. when removing a tumor. The possibility to solvethis with a motion tracking algorithm such as the onepresented remains to be analyzed. In the case of purepalpation, the online modeling already now providesvaluable information.

The integration of the presented modules intothe MiroSurge platform was possible without majorchanges in the existing control schemes presented in[47]. The achieved modularity facilitates e.g. compar-isons of the presented motion tracking algorithm withalternative approaches as presented in [15] Thus, theMiroSurge system provides a valuable platform for thedevelopment and evaluation of new advanced featuresin robotic telesurgery.

Acknowledgments

This work is supported by the AccuRobAs projectunder the 6th Framework Programme (FP6) of theEuropean Union.

References

[1] M. Altomonte, D. Zerbato, D. Botturi and P. Fiorini, Sim-ulation of deformable environment with haptic feedback ongpu, in Intelligent Robots and Systems, 2008. IROS 2008.IEEE/RSJ International Conference on, 2008, pp. 3959–3964.

[2] T. Bader, A. Wiedermann, K. Roberts and U.D. Hanebeck,Model-based motion estimation of elastic surfaces for mini--mally invasive cardiac surgery, in Proceedings of IEEEConference on Intelligent Robots and Systems IROS 07, SanDiego, USA, 2007, pp. 871–876.


[3] A. Bartoli and A. Zisserman, Direct estimation of non-rigidregistrations, in Proceedings of the 15th British MachineVision Conference, vol. 2, Kingston, UK, 2004, pp. 899–908.

[4] A. Bartoli, M. Perriollat and S. Chambom, Generalized thin-plate spline warps, in Proceedings of IEEE Conference onComputer Vision and Pattern Recognition (CVPR 07), Min-neapolis, USA, 2007, pp. 1–8.

[5] C. Basdogan, M. Sedef, M. Harders and S. Wesarg, VR-basedsimulators for training in minimally invasive surgery, IEEEComputer Graphics and Applications 27 (2007), 54–66, ISSN0272-1716.

[6] K. Bathe. Finite Element Procedures, Prentice Hall, 1995,ISBN 9780133014587.

[7] S. Benhimane and E. Malis, Real-time image-based track-ing of planes using efficient second-order minimization, inProceedings of IEEE Conference on Intelligent Robots andSystems IROS ’04, Sendai, Japan, 2004, pp. 943–948.

[8] S. Benhimane and E. Malis, Homography-based 2d visualtracking and servoing, International Journal of ComputerVision 267 (2007), 661–676.

[9] C. Cavusoglu, J. Rotella, W.S. Newman, S. Choi, J. Ustin andS.S. Sastry, Control algorithms for active relative motion can-celling for robotic assisted off-pump coronary artery bypassgraft surgery, In Proceedings of the 12th International Con-ference on Advanced Robotics (ICAR), Seattle, USA, 2005,pp. 431–436.

[10] Anirudh Choudhary, N. Moretto, F. Pizzorni Ferrarese andG.A. Zamboni, An entropy based multi-thresholding methodfor semi-automatic segmentation of liver tumors, The MIDASJournal -Grand Challenge Liver Tumor Segmentation 2008MICCAI Workshop, 2008.

[11] Force Dimension, The omega haptic master device,[web page], 2010. URL http://www.forcedimension.com.[Accessed on February 2nd, 2010].

[12] Y.C. Fung, Biomechanics -Mechanical Properties of LivingTissues, Springer-Verlag, second edition, 1993.

[13] J. Georgii, F. Echtler and R. Westermann, Interactive simu-lation of deformable bodies on GPUs, in In SimVis 05, SCSPublishing House e.V, 2005, pp. 247–258.

[14] M. Gröger, W. Sepp, T. Ortmaier and G. Hirzinger,Reconstruction of image structure in presence of specularreflections, in B. Radig and S. Florczyk, editors, PatternRecognition, Proc. 23rd DAGM Symposium, volume 2191 ofLNCS, Munich, Germany, Springer, 2001, pp. 53–60.

[15] M. Gröger, T. Ortmaier, W. Sepp and H. Gerd, Tracking localmotion on the beating heart, in SPIE medical imaging, editor,Visualization, Image-Guided Procedures and Display, vol-ume 4681, SPIE, 2002, pp. 233–241.

[16] U. Hagn, T. Ortmaier, R. Konietschke, B. Kübler, U. Seibold,A. Tobergte, M. Nickl, S. Jörg and G. Hirzinger, Tele--manipulator for remote minimally invasive surgery, IEEERobotics and Automation Magazine 154 (2008), 28–38, ISSN1070–9932.

[17] U. Hagn, R. Konietschke, A. Tobergte, M. Nickl, S. Jörg, G.Passig, M. Gröger, U. Seibold, L. Le-Tien, S. Jörg, B. Kübler,F. Fröhlich, A. Albu-Schäffer, A. Nothhelfer, F. Hacker, M.Grebenstein and G. Hirzinger, DLR MiroSurge: a versatilesystem for research in endoscopic telesurgery, InternationalJournal of Computer Assisted Radiology and Surgery, 5(2009), 183–193.

[18] James and D.K. Pai, Dyrt: dynamic response textures for realtime deformation simulation with graphics hardware, in SIG-

GRAPH’02: Proceedings of the 29th annual conference onComputer graphics and interactive techniques, New York,NY, USA, ACM, pp. 582–585, ISBN 1-58113-521-1 2002.

[19] Keiser, B. Adams, D. Gasser, P. Bazzi, P. Dutre and M. Gross,A unified lagrangian approach to solid-fluid animation, In InSymposium on Point-Based Graphics, Eurographics Associ-ation, 2005, pp. 125–133.

[20] R.M. Koch, M.H. Gross, F.R. Carls, D.F. von Büren,G. Fankhauser and Y.I.H. Parish, Simulating facialsurgery using finite element models, Computer Graph-ics, 30 (Annual Conference Series), 1996, 421–428, URLciteseer.ist.psu.edu/koch96simulating.html.

[21] R. Konietschke, Planning of Workplaces with MultipleKinematically Redundant Robots, Technische UniversitätMünchen, PhD Thesis, Munich, Germany, 2007.

[22] R. Konietschke, A. Busam, T. Bodenmüller, T. Ortmaier, M.Suppa, J. Wiechnik, T. Welzel, G. Eggers, G. Hirzinger and R.Marmulla, Accuracy Identification of Markerless Registrationwith the DLR 3D-Modeller in Medical Applications, in 6.Jahrestagung der Deutschen Gesellschaft für Computer-undRoboterassistierte Chirurgie CURAC, Karlsruhe, Germany,2007, 11–13.

[23] N. Langrana, G. Burdea, J. Ladeji and M. Dinsmore, Humanperformance using virtual reality tumor palpation simulation,Computers & Graphics, Haptic Displays in Virtual Environ-ments and Computer Graphics in Korea, 214, 1997, 451–458,ISSN 0097-8493.

[24] W. Lau, N.A. Ramey, J.J. Corso, N.V. Thakor and G.D. Hager,Stereo-based endoscopic tracking of cardiac surface deforma-tion, In Medical Image Computing and Computer-AssistedIntervention (MICCAI’04), volume 3217 of Lecture Notesin Computer Science (LNCS), Saint Malo, France, Springer,2004, pp. 494–501.

[25] L. Le-Tien, A. Albu-Schäffer and G. Hirzinger, MIMO statefeedback controller for a flexible joint robot with strong jointcoupling, in Proceedings of the IEEE International Confer-ence on Robotics and Automation ICRA, Roma, Italy, DOI:10.1109/ROBOT.2007.364065, 2007, pp. 3824–30.

[26] L. Le-Tien, A. Albu-Schäffer, A. De Luca and G. Hirzinger,Friction observer and compensation for control of robots withjoint torque measurement, in IROS, 2008, pp. 3789–3795.

[27] Q. Li and K.-M. Lee, An adaptive meshless method for mod-eling large mechanical deformation and contacts, Roboticsand Automation, 2007 IEEE International Conference on, pp.1207–1212, ISSN 1050-4729 2007.

[28] J. Lim and M. Yang, A direct method for modeling non-rigidmotion with thin plate spline. In Proceedings of IEEE Con-ference on Computer Vision and Pattern Recognition CVPR’05, Washington, USA, 2005, pp. 1196–1202.

[29] Y.-J. Lim, J. Hu, C.-Y. Chang and N. Tardella, Soft tis-sue deformation and cutting simulation for the multimodalsurgery training, In CBMS’06: Proceedings of the 19th IEEESymposium on Computer-Based Medical Systems, Washing-ton, DC, USA, IEEE Computer Society, 2006, pp. 635–640,ISBN 0-7695-2517-1.

[30] B. Lo, A.J. Chung, D. Stoyanov, G. Mylonas and G-Z Yang,Real-time intra-operative 3d tissue deformation recovery, InProceedings of IEEE International Symposium on BiomedicalImaging ISBI 08, Paris, France, 2008a, pp. 1387–1390.

[31] B. Lo, M. Scarzanella, D. Stoyanov and G. Yang, Belief prop-agation for depth cue fusion in minimally invasive surgery, inMedical Image Computing and Computer-Assisted Interven-

http://www.forcedimension.comciteseer.ist.psu.edu/koch96simulating.html


tion MICCAI ’08, Lecture Notes in Computer Science LNCS,New York, USA, Springer, 2008, pp. 104–112.

[32] E. Malis, An efficient unified approach to direct visual track-ing of rigid and deformable surfaces, in Proceedings of IEEEConference on Intelligent Robots and Systems IROS ’07, SanDiego, USA, 2007, pp. 2729–2734.

[33] J. Mosegaard, P. Herborg and T.S. Sørensen, A gpu acceler-ated spring mass system for surgical simulation, in MedicineMeets Virtual Reality 13: The Magical Next Becomes theMedical Now, 2005, pp. 342–348.

[34] M. Müller, B. Solenthaler, R. Keiser and M. Gross, Particle-based fluid-fluid interaction, in SCA ’05: Proceedings of the2005 ACM SIGGRAPH/Eurographics symposium on Com-puter animation, New York, NY, USA, ACM, 2005, pp.237–244, ISBN 1-7695-2270-X.

[35] D.G. Murphy, B.J. Challacombe and A.J. Costello, OutcomesAfter Robot-assisted Laparoscopic Radical Prostatectomy,Asian Journal of Andrology 11 (2009), 94–99.

[36] T. Ortmaier, M. Gröger, D.H. Boehm, V. Falk and G.Hirzinger, Motion estimation in beating heart surgery,IEEE Transactions on Biomedical Engineering 5210 (2005),1729–1740.

[37] R. Osborne, J. Samosky, A. Sawada, S. Gibson, C. Fyock, E.Grimson, T. Kanade, R. Kikinis, H. Lauer, N. Mckenzie, A.Mor, S. Nakajima and H. Ohkami, Volumetric object model-ing for surgical simulation, Medical Image Analysis 2 (1998),121–132.

[38] R. Richa, P. Poignet and C. Liu, 3D motion tracking for beat-ing heart surgery, International Journal of Robotics Research29(2-3) (2010), 218–230.

[39] R. Richa, A. Bo and P. Poignet, Beating heart motion pre-diction for robust visual tracking, ICRA’10: 2010 IEEEInternational Conference on Robotics and Automation, USA,2010, pp. 4579–4584.

[40] R. Richa, A. Bo and P. Poignet, Towards Robust 3D VisualTracking For Motion Compensation in Beating Heart Surgery,Medical Imaging Analysis 15(3) (2011), 302–315.

[41] G. Silveira and E. Malis, Real-time visual tracking underarbitrary illumination changes, in Proceedings of IEEEConfer-ence on Computer Vision and Pattern RecognitionCVPR’07, Minneapolis, USA, 2007, pp. 1–6.

[42] D. Stoyanov and G.Z. Yang, Stabilization of image motion forrobotic assisted beating heart surgery, in Medical Image Com-puting and Computer-Assisted Intervention (MICCAI’07),

volume 4791 of Lecture Notes in Computer Science (LNCS),Brisbane, Australia, Springer, 2007, pp. 417–424.

[43] D. Stoyanov, A. Darzi and G. Yang, A practical approachtowards accurate dense 3D depth recovery for roboticlaparo-scopic surgery, Computer Aided Surgery, 410 2005a,199–208.

[44] D. Stoyanov, G.P. Mylonas, F. Deligianni, A. Darzi and G.Z.Yang, Soft-tissue motion tracking and structure estimation forrobotic assisted mis procedures, In Medical Image Comput-ing and Computer-Assisted Intervention MICCAI’05, volume3750 of Lecture Notes in Computer Science (LNCS), PalmSprings, USA, Springer, 2005b, pp. 139–146.

[45] R.H. Taylor and D. Stoianovici, Medical Robotics inComputer-Integrated Surgery, IEEE Transactions on Roboticsand Automation 195 (2003), 765–781.

[46] A. Thakral, J. Wallace, D. Tomlin, N. Seth and N.V.Thakor, Surgical motion adaptive robotic technology s.m.a.r.t:Taking the motion out of physiological motion, In Medi-cal Image Computing and Computer-Assisted Intervention(MICCAI’01), Lecture Notes in Computer Science (LNCS),Utrecht, The Netherlands, Springer, 2001, pp. 317–325.

[47] A. Tobergte, R. Konietschke and G. Hirzinger, Planning andcontrol of a teleoperation system for research in minimallyinvasive robotic surgery, in ICRA, 2009, pp. 4225–4232.

[48] A.L. Trejos, S.E. Salcudean, F. Sassani and S. Lichten-stein, On the feasibility of a moving support for surgeryon the beating heart, in Medical Image Computing andComputer-Assisted Intervention (MICCAI’99), volume 1679of Lecture Notes in Computer Science (LNCS), Cambridge,UK, Springer, 1999, pp. 1088–1098.

[49] M. Visentini-Scarzanella, G.P. Mylonas, D. Stoyanov andG. Yang, i-brush: A gaze-contigent virtual paintbrush fordense 3d reconstruction in robotic assisted surgery, in Med-ical Image Computing and Computer-Assisted Intervention(MICCAI’09), Lecture Notes in Computer Science (LNCS),London, UK, Springer, 2009, pp. 353–360.

[50] W. Wu and P.A. Heng, A hybrid condensed finite elementmodel with gpu acceleration for interactive 3d soft tissuecutting: Research articles, Comput. Animat. Virtual Worlds,153-4 (2004), 219–227, ISSN 1546-4261.

[51] D. Zerbato, S. Galvan, and P. Fiorini, Calibration of massspring models for organ simulations, in Intelligent Robots andSystems, 2007. IROS 2007. IEEE/RSJ International Confer-ence on, 29 2007-Nov. 2 2007.

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttp://www.hindawi.com Volume 2010

RoboticsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014


Active and Passive Electronic Components

Control Scienceand Engineering

Journal of



RotatingMachinery


Hindawi Publishing Corporation http://www.hindawi.com

Journal ofEngineeringVolume 2014

Submit your manuscripts athttp://www.hindawi.com

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

SensorsJournal of


Modelling & Simulation in EngineeringHindawi Publishing Corporation http://www.hindawi.com Volume 2014


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation



DistributedSensor Networks


Documents

Integration of new features for telerobotic surgery into the …downloads.hindawi.com/journals/abb/2011/635951.pdf · 2019. 7. 31. · Applied Bionics and Biomechanics 8 (2011) 253–265