Medical Image Analysis 16 (2012) 38–49

Contents lists available at ScienceDirect

Medical Image Analysis

journal homepage: www.elsevier .com/locate /media

Registration of 3D trans-esophageal echocardiography to X-ray fluoroscopyusing image-based probe tracking

Gang Gao a,⇑, Graeme Penney a, Yingliang Ma a, Nicolas Gogin b, Pascal Cathier b, Aruna Arujuna a,Geraint Morton a, Dennis Caulfield d, Jaswinder Gill d, C. Aldo Rinaldi d, Jane Hancock d, Simon Redwood d,Martyn Thomas d, Reza Razavi a, Geert Gijsbers c, Kawal Rhode a

a Divsion of Imaging Sciences & Biomedical Engineering, King’s College London, UKb Medisys Research Group, Philips Healthcare, Paris, Francec Business Unit iXR/Business Program EP, Philips Healthcare, Best, The Netherlandsd Department of Cardiology, Guy’s & St. Thomas’ Hospitals NHS Foundation Trust, London, UK

a r t i c l e i n f o a b s t r a c t

Article history:Received 22 June 2010Received in revised form 20 April 2011Accepted 4 May 2011Available online 12 May 2011

Keywords:Image registrationCardiac interventionTrans-esophageal echocardiographyX-ray fluoroscopy

1361-8415/$ - see front matter � 2011 Elsevier B.V. Adoi:10.1016/j.media.2011.05.003

⇑ Corresponding author. Tel.: +44 02071888376.E-mail address: gaogang@gmail.com (G. Gao).

Two-dimensional (2D) X-ray imaging is the dominant imaging modality for cardiac interventions. How-ever, the use of X-ray fluoroscopy alone is inadequate for the guidance of procedures that require soft-tissue information, for example, the treatment of structural heart disease. The recent availability ofthree-dimensional (3D) trans-esophageal echocardiography (TEE) provides cardiologists with real-time3D imaging of cardiac anatomy. Increasingly X-ray imaging is now supported by using intra-procedure3D TEE imaging. We hypothesize that the real-time co-registration and visualization of 3D TEE and X-ray fluoroscopy data will provide a powerful guidance tool for cardiologists. In this paper, we proposea novel, robust and efficient method for performing this registration. The major advantage of our methodis that it does not rely on any additional tracking hardware and therefore can be deployed straightfor-wardly into any interventional laboratory. Our method consists of an image-based TEE probe localizationalgorithm and a calibration procedure. While the calibration needs to be done only once, the GPU-accel-erated registration takes approximately from 2 to 15 s to complete depending on the number of X-rayimages used in the registration and the image resolution. The accuracy of our method was assessed usinga realistic heart phantom. The target registration error (TRE) for the heart phantom was less than 2 mm.In addition, we assess the accuracy and the clinical feasibility of our method using five patient datasets,two of which were acquired from cardiac electrophysiology procedures and three from trans-catheteraortic valve implantation procedures. The registration results showed our technique had mean registra-tion errors of 1.5–4.2 mm and 95% capture range of 8.7–11.4 mm in terms of TRE.

� 2011 Elsevier B.V. All rights reserved.

1. Introduction

Minimally-invasive cardiovascular interventions are carried outfor the diagnosis and treatment of a broad range of cardiovasculardiseases. These types of procedures are increasingly popular whencompared to their more invasive counterparts because there is lessmorbidity to the patient with similar clinical outcomes of success.Examples of these procedures include those carried out for the re-pair of structural heart disease and cardiac electrophysiology (EP)procedures. The interventional devices, for example, catheters,are designed to be X-ray visible and can be seen throughout thepart of their length that lies in the X-ray field of view (FOV).Two-dimensional (2D) X-ray imaging is the dominant imagingmodality for guiding cardiac interventions. Typically, imaging can

ll rights reserved.

be performed at high frame rates (up to 30 frames per second)and therefore the cardiac motion and the motion of interventionaldevices do not cause significant motion artifacts in the acquiredimages. However, the use of X-ray fluoroscopy alone is inadequatefor the guidance of procedures that require soft-tissue information,for example, the treatment of structural heart disease (Silverstryet al., 2009). In addition, exposing patients, especially pediatric pa-tients with congenital defects, to ionizing radiation carries a signif-icant risk (Kovoor et al., 1998; Modan et al., 2000). On-goingresearch is actively seeking to reduce the use of, or even replace,X-ray fluoroscopy in cardiac interventional procedures, especiallyfor pediatrics (Razavi et al., 2003). Pre-operatively acquired mag-netic resonance imaging (MRI) and computed tomography (CT)have been used with X-ray fluoroscopy to improve the guidanceof cardiac catheterization procedures (Rhode et al., 2003, 2005;Yu et al., 2005; De Buck et al., 2005; Sra et al., 2007). The imageregistration of pre-procedure MRI/CT and intra-procedure X-ray

G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 39

can be achieved either by using prior calibration (Rhode et al.,2003, 2005; Yu et al., 2005), manual registration (De Buck et al.,2005), or automatic approaches (Sra et al., 2007). Recently, theuse of three-dimensional (3D) rotational X-ray angiography (3DRXA) (Knecht et al., 2010; Manzke et al., 2010) has been shownto be useful for cardiac interventional guidance. This imagingmodality is perfectly suited to efficient clinical work flow sincethe 3D image data are acquired just prior to the intervention withthe patient lying on the same X-ray table. Therefore, the registra-tion of the 3D data to the 2D fluoroscopy data is implicit. However,with the limitation of the current technology the image quality andFOV of 3D RXA are considerably lower than those of MRI and CT.

The use of pre-procedural MR/CT and 3D RXA imaging will pro-duce roadmap images that are static and do not update with theintra-procedural situation. Additional steps are required to com-pensate for intra-procedural deformations, for example, thosecaused by respiratory motion of the heart (King et al., 2009). Incontrast to MR and CT imaging, 3D echocardiography (echo) is areal-time imaging modality that can be readily used in the catheterlaboratory environment. Because of its relatively low image qualityand small FOV, 3D echo has been used as an intermediate imagingmodality to register X-ray fluoroscopy with high resolution imag-ing modalities such as MRI and CT (King et al., 2010). Although pre-vious works have successfully registered 3D echo with otherimaging modalities such as MRI (King et al., 2010; Roche et al.,2001; Penney et al., 2004; Wein et al., 2009), the registration be-tween echo and X-ray fluoroscopy still relied on prior calibrationand tracking. With recently emerging technologies such as 3Dtrans-esophageal echo (TEE), the image quality and resolution ofecho has improved considerably. In comparison to trans-thoracicecho, TEE has unique access to structures such as the aorta, thevalves of the heart and both atria. This makes TEE an ideal solutionfor the guidance of cardiac interventions. Recently, the clinical fea-sibility of 3D TEE was evaluated for guiding a variety of cardiacinterventional procedures including atrial septal defect closure,patent foramen ovale closure, mitral valve/aortic valve repair andEP procedures (Barker et al., 2008; Mackensen et al., 2008). TEE ap-pears to be a promising imaging modality for cardiac interventionsand its routine clinical use is rapidly increasing. It is already well-established in the clinical work flow of several interventions, suchas minimally-invasive valve implantation procedures.

Combining the soft-tissue information from TEE with the excel-lent device visualization of X-ray fluoroscopy could prove to be apowerful combination in the catheter laboratory. The purpose ofthis study was to develop a practical technique to combine 3DTEE and X-ray fluoroscopy images for the guidance of cardiac cath-eterization procedures. A fast and robust image registration of 3DTEE and X-ray fluoroscopy was developed for this purpose. Previousapproaches relied on the use of an electromagnetic (EM) trackingdevice (Gao et al., 2009; Jain et al., 2009). However, the use of theEM tracker has several disadvantages including the requirementof additional hardware, the requirement of modifications to theultrasound probe, and the sensitivity of these systems to metallicinterference. Neither of the studies reported in (Jain et al., 2009;Gao et al., 2009) involved clinical evaluations but were limited tophantoms, therefore, the performance of EM tracking in the routineclinical work flow is unclear. Our method was based upon an im-age-based 2D–3D registration algorithm. It does not rely on track-ing hardware or modification to the TEE probe and therefore canbe deployed straightforwardly to any cardiac catheterization labo-ratory. Preliminary results were reported in (Gao et al., 2010). Inthis paper, we will describe a more thorough study to examinethe performance of the 2D–3D registration algorithm and the over-all TEE and X-ray image registration using both phantom and clin-ical data. Clinical data (n = 5), which were acquired from twodifferent types of cardiac intervention procedures, including car-

diac EP procedures and trans-catheter aortic valve implantations(TAVI), were used to examine the clinical feasibility of the proposedalgorithm. Although all the clinical data were currently processedoff-line, a clinical workflow was proposed to demonstrate howour technique will be used live in the clinical environment.

2. Methods

2.1. Localization of the TEE probe

Our method is based on the localization of the TEE probe. Sincethe addition of tracking devices, such as an EM tracker, is not idealfor the clinical cardiac catheterization environment, the key chal-lenge in our method is to design an efficient and robust image-based TEE probe tracking algorithm. Fig. 1a and b shows the 3DTEE probe used in this study and an X-ray image acquired duringan aortic valve implant procedure, clearly showing the TEE probe.Our hypothesis is that by using one or more X-ray images, the po-sition and the orientation of the TEE probe can be automaticallydetermined from the X-ray data.

2.1.1. 3D reconstruction of the TEE probeAlthough the TEE probe is designed to be flexible, its transducer

is encapsulated in a rigid head. For simplicity, the rigid head of theTEE probe is referred to as ‘‘the TEE probe’’ in the following text.The first step of our localization algorithm is to reconstruct a pre-cise 3D model of the TEE probe. Our approach is to use a nano-CTsystem (nano-Pet™/CT, Mediso Ltd., Budapest, Hungary) which iscapable of reconstructing ultra-high-resolution 3D volumes.Fig. 1c shows the nano-CT reconstructed TEE probe (matrix:342 � 342, resolution = 0.2 � 0.2 � 0.2 mm3). The 3D reconstruc-tion of the TEE probe does not need to be repeated unless a differ-ent model of TEE probe is used.

2.1.2. 2D–3D image registrationIn order to determine the position of the TEE probe in C-arm

space (i.e. the local 3D coordinate system of the X-ray system),the nano-CT volume of the TEE probe was registered to the X-rayimages using a 2D–3D image registration algorithm. Intensity-based 2D–3D image registration has been extensively studied(Penney et al., 1998; Hipwell et al., 2003; van de Kraats et al.,2005; Turgeon et al., 2005). It has been used to register pre-opera-tive CT or MR volume of bones, blood vessels and medical devicesto intra-operative X-ray images (Hipwell et al., 2003; Turgeon etal., 2005). Fig. 2 shows a typical workflow of an intensity-based2D–3D registration algorithm. During the registration process,the algorithm repeatedly repositions the 3D volume in space andcompares its projection, called the digitally reconstructed radiograph(DRR), with one or multiple X-ray image(s). At each iteration, animage comparison metric is used to calculate the similarity be-tween the X-ray image(s) and the DRR. The three translation andthree rotation parameters are changed according to the similaritymeasurement. The registration process continues until the similar-ity between the X-ray image(s) and the DRR is maximized.

In this study, we implemented an intensity-based 2D–3D regis-tration algorithm similar to the one described in (Penney et al.,1998). This work shows gradient difference (GD) and patternintensity (PI) are more robust and accurate than other intensity-based similarity measurements, such as mutual information orcross correlation. Both GD and PI are largely unaffected by thepresence of soft tissue and thin line structures, such as guide wires,stents and other interventional devices that appeared in the X-rayfluoroscopy. In this study, GD was used as the similarity measure-ment function in our 2D–3D registration algorithm. A classic Pow-ell function was used as the optimizer. The speed of the original

Fig. 1. (a) The 3D TEE probe used (X7-2t, Philips Healthcare, Andover, Boston, USA); (b) A typical X-ray fluoroscopy image of the TEE probe; (c) A high-resolution 3D nano-CTscan reveals the contours and the internal structure of the TEE transducer. The 3D nano-CT volume was used to localize the 3D position of the TEE probe in X-ray space.

Fig. 2. The workflow of an intensity-based 2D–3D registration algorithm.

40 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

algorithm described in (Penney et al., 1998) was relatively slow be-cause the generation of DRRs is computationally expensive. Itcould take minutes or longer to register the nano-CT volume to astandard X-ray image. Our implementation overcomes this issueby taking advantage of the latest GPU technology. The DRR recon-struction time was reduced to less than 10 ms for each estimatedpose. By using a standard computer workstation equipped withan Intel Quad-Core CPU (2.66 GHz), 4 GB RAM and an NVIDIA Ge-Force GTX 280 graphics card, the overall registration time betweenthe nano-CT volume and two standard X-ray images is from 1 to15 s, depending on the initial probe position and the size and num-ber of X-ray images used. The evaluation of registration speed willbe presented in Sections 3.2.1 and 4.1.2.

It is well acknowledged in previous studies that 2D–3D registra-tion using a single X-ray image can potentially introduce consider-able registration error in the direction of the X-ray beam. Tominimize such error, two X-ray images acquired from different an-gles must be used. Our 2D–3D registration algorithm can processone or multiple X-ray images simultaneously depending on therequirement of the study. In-depth discussion related to this topicwill be presented in Sections 4.1 and 5.

The authors acknowledge that the registration of TEE and X-rayis a form a 2D–3D registration. However for simplicity and clarity,the ‘‘2D–3D registration’’ mentioned in the following text refersspecifically to the intensity-based 2D–3D image registration ofnano-CT and X-ray.

2.2. Image registration of 3D TEE and X-ray fluoroscopy

The transformation matrix, TUS_img?X-ray_img, which transformsfrom 3D TEE image space to 2D X-ray image space consists of a ri-gid body transformation matrix, Trigid, and a projection matrix, Tproj,

TUS img!X�ray img ¼ TprojTrigid: ð1Þ

The projection matrix, Tproj, transforms from 3D C-arm space, i.e.the local coordinate system of the X-ray system, to 2D X-ray imagespace. This can be calculated by using the intrinsic parameters ofthe X-ray system that are determined from a system calibrationand are available from the DICOM image header (Hawkes et al.,1987). Trigid can be decomposed into two matrices

TUS img!X�ray img ¼ TprojTtracked!C-armTUS img!tracked ð2Þ

where in our case, Ttracked?C-arm is the transformation matrix fromthe 3D tracked coordinate system, i.e. the nano-CT space, to 3D C-

arm space. This matrix is generated by the 2D–3D registration algo-rithm that positions the nano-CT volume in C-arm space. TUS_img?

tracked relates the position of the 3D TEE images with the positionof the TEE probe in nano-CT space. This is the TEE probe calibrationmatrix and can be calculated pre-procedurally using a specificallydesigned calibration phantom.

2.3. TEE probe calibration

The TEE probe calibration procedure aims to determineTUS_img?tracked (Mercier et al., 2005). The calibration phantomconsists of a 9-L water tank and two thin metal wires. Nine metallandmarks which were visible in both X-ray and ultrasound wereplaced on the wires. The TEE probe was rigidly fixed beneath thewires during data acquisition. X-ray images were acquired from leftanterior oblique (LAO) 45�, right anterior oblique (RAO) 45� andposterio-anterior (PA) projections using a Philips Allura Xper FD10

Fig. 3. The proposed workflow of the X-ray and TEE image registration in clinicalprocedures.

Fig. 4. The heart phantom experiment was performed in the catheter laboratory.The picture shows the system setup of the experiment.

G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 41

C-arm X-ray system, which has an internal mechanism to track theC-arm position in real-time. Simultaneously an echo volume was ac-quired in full volume mode, giving the maximal volume coveragepossible with the TEE probe. The following steps were adopted todetermine TUS_img?tracked: (a) a quick manual registration was per-formed to align the nano-CT volume with the X-ray images acquiredfrom PA & LAO; and (b) the automatic 2D–3D registration algorithmdescribed in Section 2.2 was then utilized to refine the alignmentsimultaneously in both views. This registration ensured an accuratelocalization of the TEE probe in 3D C-arm space, i.e. determination ofTtracked?C-arm. The third X-ray image, which was acquired from RAO45� was used to confirm the accuracy of the TEE probe localization.

The 3D positions of the nine landmarks, PphantomUS, were identi-fied manually from the TEE image data. The landmarks were alsoclearly visible in the X-ray images. By manually defining the 2D posi-tion of the landmarks in the PA and LAO 45� X-ray images, their 3Dpositions in C-arm space, PphantomC-arm, could be reconstructed usingback-projection (Hawkes et al., 1987). The calibration procedure wasrepeated for three different probe positions. A classic hill-climbingoptimization algorithm was employed to find TUS_img?tracked byminimizing the Euclidean distance error, e, given by

e ¼ TUS img!trackedPphantomUS � T�1tracked!C-armPphantomC-arm

������ ð3Þ

In order to validate the accuracy of the calibration, a further 2TEE volumes were acquired of the calibration phantom along withX-ray images in the PA and LAO 45� views. For these data, we usedEq. (3) to compute the mean 3D Euclidean distance errors intracked nano-CT space.

2.4. Clinical workflow

The objective of this study was to develop a practical solution tointegrate X-ray and 3D echo for the guidance of cardiac interven-tional procedures. We propose a clinical workflow to demonstratehow the proposed TEE and X-ray registration algorithm can beused in clinical environments. The workflow consists of seven com-ponents (Fig. 3), some of which are currently manual but could beautomated in the future, for example, the initial 2D–3D registra-tion and the failure detection for the probe localization. The probetracking component, which is designed to compensate the jitteringmotion of the TEE probe caused by cardiac and respiratory motion,is not available at present. It is an open research topic. One solutionwould be to speed up the intensity-based 2D–3D registration byincorporating more sophisticated implementations such as multi-ple-resolution techniques and advanced GPU computing. More de-tails about the jittering motion of the TEE probe will be discussedin Section 3.2.3. Technically, the 2D–3D registration algorithm canwork either with mono-plane or bi-plane X-ray. The workflow doesnot restrict the number of X-ray images used in the 2D–3D regis-tration. The decision on whether mono-plane or bi-plane X-rayimages will be used should be made based on the clinical require-ments and how the registration results will be utilized.

3. Experiments

3.1. Imaging equipment and data acquisition

Experimental data were acquired from the TEE probe calibra-tion phantom, a realistic heart phantom and five patients undergo-ing cardiac interventional procedures.

3.1.1. Realistic heart phantomWe evaluated the accuracy of our method using a realistic heart

phantom (Ultrasound Heart Phantom, Computerized Imaging Ref-erence Systems, Inc., Virginia, USA). The phantom has completely

anthropomorphic external and internal anatomy including left/right ventricles, left/right atria and the valves (Fig. 4).

For TEE acquisition, we used an iE33 3D real-time echo systemwith an X7-2t 3D TEE probe (Philips Healthcare, Andover, Boston,USA). The TEE probe was placed on the acoustic surface of the heartphantom. Six TEE volumes were acquired to cover differentsections of the phantom by varying the TEE probe position. NineX-ray and MRI-visible markers (Multi-modality RadiographicMarker, IZI Medical Products, Baltimore, USA) were placed on thesurface of the phantom. The MR volume of the heart phantomwas acquired a day before the TEE and X-ray scans in the catheterlaboratory using a Philips Achieva 1.5T MR scanner. The MR proto-col consisted of a 3D BTFE scan with the following parameters:512 � 512 matrix, 180 slices, resolution = 0.96 � 0.96 � 1.0 mm3,TR = 8.42 ms, TE = 4.2 ms, flip angle = 90�. Between the MRI scanand the TEE scan, the heart phantom was carefully stored in orderto avoid displacement of the markers. For X-ray image acquisition,

42 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

we used a Philips Allura Xper FD10 C-arm X-ray system, the sameX-ray system used for the TEE probe calibration.

3.1.2. Cardiac EP proceduresWe collected data from two cardiac EP procedures. Both of the

patients had left atrial flutter and were under general anesthesia(GA) during the procedures. For the first patient, TEE and X-raydata were acquired after two deca-polar catheters were insertedinto the right atrium (RA), one forming a loop along the endocar-dial surface of RA and the other inserted into the coronary sinus(CS). Both catheters were visible in the TEE volume. For the secondpatient, a trans-septal puncture was performed to gain access tothe left atrium (LA). TEE volumes were acquired after a lasso cath-eter and an ablation catheter were inserted into LA. The movementof the C-arm was limited by other equipment such as the life sup-port system and the ultrasound scanner. X-ray images covering 4–5 cardiac cycles were acquired from PA and either RAO 30� or LAO30� projections. Cardiac and respiratory phases were estimatedusing the motion of the interventional devices such as the cathe-ters (Brost et al., 2010; Klemm et al., 2007). X-ray images acquiredduring the same cardiac and respiratory phases were used in the2D–3D registration.

3.1.3. Trans-catheter aortic valve implant proceduresTAVI data were collected from three patients with degenerative

aortic stenosis. All the three procedures were performed in a GEcatheter laboratory equipped with a GE Innova 2100IQ C-arm X-ray system. Similar to the Philips Allura Xper FD10, this GE X-raysystem can precisely track its C-arm position automatically. TheTEE data were acquired using the same echo system and TEE probedescribed above. In the first two TAVI cases, the replacementvalves were delivered using trans-femoral approach. In the thirdTAVI case, the replacement valve was delivered using trans-apicalapproach. X-ray images and TEE volumes were acquired in thesame way as the data acquisition in the EP procedures after thereplacement valve reached the deployment site.

3.2. Evaluation of the nano-CT and X-ray image registration

By using our method, a successful TEE and X-ray image registra-tion depends on a fast and robust 2D–3D nano-CT and X-ray imageregistration algorithm. In this experiment, we evaluate the robust-ness and the speed of the 2D–3D registration algorithm using stan-dard methodologies presented in previously published literature(van de Kraats et al. 2005).

3.2.1. Experiment 1 – Capture range and speed of the 2D–3Dregistration

The capture range was defined as the largest starting misalign-ment at and below which registration was successful 95% of the time(Huang et al., 2009; Shekhar and Zagrodsky, 2002). To evaluate thecapture range, a reference transformation matrix was introducedfor each of the testing datasets. The reference transformation matrixwas prepared by the following procedures:

(a) the automatic 2D–3D registration, as described in Section2.1, was employed to generate a starting position to alignthe nano-CT volume with bi-plane X-ray images;

(b) the registration was then refined using manual expert inter-action allowing all six degrees of freedom during carefulvisual examination of the result to ensure the closest possi-ble match;

(c) steps (a) and (b) were repeated three times resulting in threetransformation matrices. The reference matrix was con-structed using the mean parameters of these three matrices.

The starting positions were created as follows:

(a) For each of the six degrees of freedom, a range was deter-mined. Within the ranges, transformation matrices weregenerated using all possible combinations of the six DOFs.The center of the rotation is the center of the nano-CT vol-ume. New positions of the nano-CT volume were generatedby transforming the original nano-CT volume using thematrices;

(b) For each of the newly generated nano-CT positions, its cor-ner point distance to the position of the original nano-CTvolume was calculated as the mean target registration error(Eq. (4)). The new nano-CT positions were sorted using theirmTRE to the original nano-CT position;

(c) Intervals were chosen for the starting position distance, forexample, 0–1 mm, 1–2 mm. For each interval, 100 newnano-CT positions were selected randomly (uniformly dis-tributed) as the starting positions.

For the calculation of mTRE, the landmarks were the eight cor-ners of the bounding box of the nano-CT volume. The testing dataused in this experiment were X-ray images acquired from the clin-ical studies. For each of the seven testing datasets, 1400 startingpositions (1 mm 6mTRE 6 15 mm) were examined to calculatethe capture range. The capture ranges using bi-plane X-ray imagesand mono-plane X-ray images were studied separately

mTRE ¼ 1n

Xn

i¼1

TregistrationPilandmark � Tref P

ilandmark

������ ð4Þ

where P denotes the landmarks defined on the eight corners of thenano-CT volume. To evaluate the registration accuracy using mono-plane X-ray images, the mean projection distance, mPD was calcu-lated (van de Kraats et al., 2005)

mPD ¼ 1n

Xn

i¼1

MregistrationPilandmark �Mref P

ilandmark

� ������� � Sdet � F ð5Þ

where Mregistration and Mref are the final perspective projectionmatrices for the registration and the reference. Sdet is the detectorsize of the X-ray system. F is the magnification factor, which canbe calculated using Eq. (6)

F ¼ Dsource!object

Dsource!imageð6Þ

Dsource?object and Dsource?image are source to object distance andsource to image distance. Sdet, Dsource?object and Dsource?image can beretrieved from either the live data stream from the X-raysystem or the header of the DICOM files. It should be noted thatDsource?object is an approximated value and therefore the meanprojection distance calculated using Eq. (5) is also an approximatedvalue.mTRE has been previously proposed and accepted as a metricto evaluate the capture range of 2D–3D registration in differentstudies (Huang et al., 2009; van de Kraats et al., 2005). However,it cannot clearly differentiate the errors in translation from theerrors in rotation. An additional step was used to examine the reg-istration error in translations (mm) and rotations (�) separately byusing the magnitude of the rotations and the translations

t ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffit2

x þ t2y þ t2

z

qand h ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffih2

x þ h2y þ h2

z

qð7Þ

where t and h are the magnitude of the translations and the rota-tions, tx, ty, tz and hx, hy, hz are the translations and rotations in x,y and z directions.

The computation time of every registration was recorded toanalyze the efficiency of our 2D–3D registration algorithm.

Fig. 6. MRI was used as a gold standard to assess the accuracy of the TEE and X-rayimage registration.

G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 43

3.2.2. Experiment 2 – Evaluation of TEE and X-ray registration errorwith regard to distance from the transducer

The 2D–3D registration error will be propagated into the overallTEE and X-ray registration and small rotational errors can poten-tially create a considerable error, especially for a target object atthe far side of the ultrasound cone. In this experiment, we exam-ined how the error produced by the 2D–3D registration algorithmaffects the overall TEE and X-ray registration accuracy.

In the previous experiment, a considerable number of registra-tions were performed. The registration can be considered as suc-cessful if the end mTRE is not greater than a pre-definedthreshold. The matrices from all the successful registrations wereused to transform a number of landmarks defined within the TEEFOV at different depths (Fig. 5). mTRE was calculated using Eq.(4) to examine the potential target error at different depths in com-parison to the reference matrix.

3.2.3. Experiment 3 – Quantitative study of the error caused byjittering motion of the TEE probe

As mentioned previously, the jittering motion of the TEE probeis unavoidable due to cardiac and respiratory motion. A real-timeprobe tracking algorithm will minimize such error. We do not in-tend to propose such a tracking algorithm in this paper. Instead,an experiment was set up to investigate the potential error causedby the jittering motion of the TEE probe. The experiment consistsof three steps:

(1) acquire X-ray images of the TEE probe during clinical proce-dures to cover at least one respiratory cycle so that the cap-tured jittering motion consists of both respiratory motionand cardiac motion;

(2) register the nano-CT volume to each of the X-ray images;(3) set the first frame of the X-ray sequence as the reference and

calculate the TEE probe motion from the second frameonwards in terms of mPD by:

mPDiðiP2Þ ¼1n

Xn

j¼1

ðMiPlandmark;j �M1Plandmark;jÞ�� �� � Sdet � F ð8Þ

Fig. 5. The FOV of the TEE is a pyramid shape structure which can be defined fromany TEE volume. Ten image planes were selected within the TEE FOV which areparallel to the transducer of the TEE probe. The interval between two image planeswas 10 mm. The landmarks defined on the four corners of the image planes wereused to examine the potential error for targets at different depths caused purely bythe error of the 2D–3D registration algorithm.

3.3. Accuracy assessment of TEE and X-ray image registration

3.3.1. Phantom studySince the heart phantom is barely visible in X-ray, the MR

images of the heart phantom were used as a reference to assessthe accuracy of the TEE and X-ray image registration. The workflowof the accuracy assessment is illustrated in Fig. 6. Firstly, the 3Dpositions of the landmarks were determined manually in the MRimage. Secondly, positions of the corresponding landmarks weredetermined manually from bi-plane X-ray images acquired fromLAO 45� and RAO 45� so that their 3D positions could be accuratelyreconstructed. A third X-ray image was acquired from PA view. Thelandmarks were projected back to all three X-ray images (LAO 45�,RAO 45� and PA) so that the 3D reconstruction accuracy can beestimated by calculating the distance between the projected land-marks and their manually defined positions. The MRI and C-armcoordinate system registration was achieved using a landmark reg-istration. To quantify the TEE and X-ray registration error, endocar-dial surfaces of the heart phantom including left ventricle (LV),right ventricle (RV), left atrium (LA), right atrium (RA) and left ven-tricular papillary muscle (LVPM) were extracted and reconstructedfrom the MR volume semi-automatically using an open sourcesoftware application ITK-Snap (Yushkevich et al. 2006). A total of16–66 points were defined manually along the endocardial borderfrom the TEE volumes. Both the MR surfaces and the TEE pointswere transformed to C-arm space. The registration error was as-sessed both visually and by calculating the mean distance betweenthe TEE points and the MR surface. Point-to-surface distances donot capture errors that are tangential to the surface but in our casethe surfaces were largely blob-like and therefore this should not af-fect the results.

It should be noted that the point-to-surface distance representsa combination of the MR to C-arm landmark registration error andTEE to C-arm registration error.

3.3.2. Clinical studyTo assess the accuracy of the clinical studies, we identify the

interventional devices (ablation catheters, trans-catheter valves)in the TEE volume and extracted their center lines manually. Thecenter lines were then projected onto the X-ray images using theTEE to X-ray registration algorithm described above. A number ofpoints were then defined along the center line of the same inter-ventional device on the X-ray image. The registration error in termof mPD was calculated using the mean distance between the pointsand the projected center line of the interventional devices. Point-to-line distances cannot capture the errors that are tangential to

Fig. 7. An example of a successful 2D–3D image registration. The X-ray images(first row) were acquired during a TAVI case from RAO 29� (left) and LAO 30� (right).The nano-CT volume was registered with and projected onto both of the images(second row).

Table 1The capture range of the 2D–3D registration. The testing data were patient datacollected during the clinical procedures.

Study PatientID

Procedure Image size Capture range (mm)

Two X-rayimages

One X-rayimage

1 1 EP 1024 � 1024 9.7 10.42 2 EP 1024 � 1024 11.4 11.23 2 EP 1024 � 1024 10.3 9.84 3 TAVI 512 � 512 10.7 10.15 3 TAVI 512 � 512 9.2 8.76 4 TAVI 512 � 512 9.6 9.47 5 TAVI 512 � 512 8.7 9.8

44 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

the line. However, several of the devices, especially the EP cathe-ters, had substantial curvature, reducing the significance of thisproblem.

4. Experimental results

4.1. Nano-CT and X-ray image registration

In this section, we present the image registration results be-tween the nano-CT volume of the TEE probe and the X-ray images.

4.1.1. Capture range and initial registrationFig. 7 shows an example of a successful registration between

the nano-CT volume and two X-ray images. Fig. 8 shows the results

Fig. 8. (a) For each of the datasets, 1400 starting positions with mTRE ranging evenly fr2D–3D image registration algorithm. (b) For all the successful registrations, landmarks werror caused by the error of the 2D–3D image registration. Presuming that the 2D–3D imaTEE and X-ray image registration for the targets defined within the TEE FOV at differen

of the capture range analysis using a randomly selected clinicaldataset. Given the success rate threshold of 2.5 mm, the capturerange of the 2D–3D registration algorithm was 9.2 mm (Fig. 8a).The error generated by most of the failed registrations (89.2%)was greater than 10 mm while most of the successful registrationscreated an error of less than 1 mm (98.7%). The success rate thresh-old can be altered between 1 mm and 10 mm without considerablychanging the capture range result. This is in-line with the previousfindings in (van de Kraats et al., 2005). Fig. 8b shows the potentialTEE and X-ray registration error propagated from the 2D–3D regis-tration at different depths of the ultrasound cone for all the suc-cessful 2D–3D registrations. In practice, the imaging targets suchas the heart valves were routinely positioned in the middle ofthe ultrasound FOV at around 5–6 cm depth. The potential mTREcaused by the 2D–3D registration at 6 cm depth is approximately1 mm. Overall more than 95% of the registrations achieved anmTRE of less than 2.5 mm. The capture range study was carriedout for each of the clinical data sets. The results are summarizedin Table 1. For all the successful registrations (mTRE 6 2.5 mm),the mean error in terms of rotations and translations are listed inTable 2. Fig. 9 shows the stating positions for all the failed registra-tions in terms of rotations and translations. It suggests only 4.2% ofthe failed registrations start within a region of 10 mm � 10�. How-ever, if the starting position of the TEE probe is within a region of12 mm � 12�, the percentage increases dramatically to 31.6%. Thisindicates that an initial approximate registration should correct

om 1 mm to 15 mm were used to examine the capture range of the intensity-basedere defined within the TEE FOV at different depths to examine the potential targetge registration was the sole error source, the red curve shows the success rate of the

t depths.

Table 2The errors of all the successful registration in terms of rotation and translation.

Translation Rotation

tx (mm) ty (mm) tz (mm) rx (�) ry (�) rz (�)

Mean err. 0.06 0.16 0.04 1.16 0.17 1.05Std. dev. 0.03 0.06 0.03 0.23 0.11 0.58Max. err. 0.47 1.07 1.06 3.13 1.47 2.46

Fig. 9. The starting positions of all the failed 2D–3D registrations in terms oftranslation and rotation. Only 4.2% of the failed registrations started from within aregion of 10 mm � 10�.

Fig. 10. (a) Comparison of 3D errors in term of mTRE for the 2D–3D registrationusing bi-plane X-ray and mono-plane X-ray. The testing data were acquired fromthe first of the TAVI procedures reported in this paper. Potentially, the 2D–3Dregistration using one X-ray image could produce considerable error in the X-rayprojection direction, reducing the overall success rate. (b) The majority of the erroris in the direction of the X-ray projection, the in-plane error mPD.is 1.8 ± 1.3 mmcompared to the overall 3D error of 6.8 ± 7.8 mm.

G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 45

the misalignment to within a 10 mm � 10� region to be subse-quently successful for automatic registration.

Technically, it is possible to register the nano-CT volume withmono-plane X-ray images. The capture range of the image registra-tion using one X-ray image is similar to those using bi-plane X-rayimages. However, it could introduce a considerable error in term ofmTRE to the target defined within the FOV of TEE (Fig. 10a). Themajority of the error occurs along the direction of the X-ray beamand does not impact considerably if the echo data is used as anoverlay image on top of the X-ray fluoroscopy as demonstrated la-ter in Figs. 13 and 14. Fig. 10b shows that the average in-plane reg-istration error in terms of mPD is 1.8 ± 1.3 mm. We do not restrictthe number of X-ray images used in our clinical workflow. Thedecision will be made based on the requirement of the interven-tional procedures. More details will be discussed in Section 5.

4.1.2. Registration speedThe computational time for the 2D–3D image registration de-

pends on several factors including the starting position of thenano-CT volume, the size of the X-ray images and the number ofthe X-ray images used in the registration. From over 30,000 regis-trations performed using different starting positions, the computa-tional time of our 2D–3D image registration was recorded and issummarized in Table 3.

4.1.3. Jittering motion errorX-ray images acquired from a TAVI procedure (Fig. 11a) and an

EP procedure (Fig. 11b) were used to assess the error caused by thejittering motion of the TEE probe. Fig. 11 shows the curves of the

jittering motion which clearly reveal the pattern of cardiac andrespiratory motion. For both cases, the jittering motion introducesa maximum error of around 4 mm in the probe localization(4.1 mm and 3.6 mm, respectively). The maximum errors contrib-uted by the rotational components of the jittering motion are1.6 mm and 1.0 mm, respectively. Although our experiment showsthe majority of the jittering motion is translational, the error canstill make a considerable impact on the accuracy of the overallTEE and X-ray registration especially if the target object locatesat the far end of the echo FOV. In the proposed clinical workflow,a real-time TEE probe tracking algorithm will minimize the errorcaused by the jittering motion. Before such an algorithm becomesavailable, physicians will need to accommodate the registration er-ror using the motion pattern of the interventional devices observedin the X-ray fluoroscopy and in the echo volume.

4.2. TEE probe calibration

The residual calibration error from Eq. (3) was 2.0 mm. By usingthe calibration matrix, the last two volumes that were not involvedin the calibration were transformed to the C-arm space as wherethe landmarks from the X-ray images. The mTREs computed inC-arm space were 4.6 ± 1.1 mm and 5.0 ± 0.8 mm. The determina-tion of the landmark positions from the TEE volume was difficultdue to the noise and the shadowing effects of ultrasound. The

Fig. 11. The TEE probe localization error caused by the jittering motion. X-ray images were acquired during (a) a TAVI procedure and (b) a cardiac EP procedure.

46 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

mTREs show not only the registration error but also the subjectiveerror in the determination of the landmark positions.

4.3. Phantom study

The 3D positions of the landmarks were reconstructed in C-armspace using two X-ray images acquired from LAP 45� and RAO 45�.The mean reconstruction error in term of mPD was much less than1 mm (0.44 mm using the image acquired from LAO 45� and0.14 mm using image from RAO 45�). The mean landmark registra-tion error for the MR to C-arm registration was 1.03 mm. Thepoint-to-surface distance errors between TEE and MRI data aresummarized in Table 4. The mean TEE to C-arm registration errorwas in a range 1.5–2.4 mm. Fig. 12 shows the MRI-derived LVand the RV surfaces registered with two TEE volumes in C-armspace.

4.4. Clinical studies

Fig. 13 shows the registration results when different catheterswere used for two patients undergoing EP procedures. At this

Fig. 12. Two TEE images and the MR volume of the heart phantom are showntransformed to C-arm space using either the proposed method or the landmarkregistration, respectively. Presuming a comparatively high accuracy for thelandmark registration, the alignment between the MR volume and the TEE imagesindicates the accuracy of the proposed method.

stage, the data acquisition of TEE and X-ray data was not synchro-nized. The end systolic phase was manually selected from theX-ray and TEE sequences to calculate the mPD shown in Table 5.However, the catheter motion (caused by the cardiac motionincluding cardiac contraction and respiratory motion) shown inthe X-ray images was more considerable than that shown in theTEE volume. This is because the catheter motion shown in TEEwas partly canceled by the motion of the TEE probe itself. At pres-ent, our 2D–3D registration algorithm is not fast enough to com-pensate for the TEE probe motion in real-time. This error will beadditive in the registration results.

Fig. 14 shows the X-ray images and a TEE volume acquired in oneof the TAVI procedures. The diameter of the TAVI catheter (18-22F or�6–7 mm) is considerably larger than the EP catheters (8F,�3 mm).Although both the TAVI catheters and the valves can be easily iden-tified from the TEE volume, it is difficult to determine the exact loca-tion of the replacement valve in the TEE volume. Fig. 14a shows theposition of the replacement valve was clearly visible in the X-rayimages. The combined image of X-ray and TEE reveals that thereplacement valve had passed through the aortic valve into the leftventricle. Without seeing the overlay image, the physician decidedto deploy the replacement valve in this sub-optimal position. Post-deployment Doppler revealed severe paravalvular leakage and poorhemodynamic performance. The motion of the mitral valve was alsoobstructed with the mis-placed aortic valve. A second replacementvalve was inserted to stop the leakage. This case highlights the po-tential clinical impact of our technique.

The registration accuracy for all the clinical cases (two EP andthree TAVI) is summarized in Table 5.

5. Discussion

5.1. Robustness and accuracy

The novelty of our method is that we employed an image-based2D–3D registration algorithm to localize the TEE probe. The perfor-mance of 2D–3D registration algorithms is usually data dependent.However, our task is more straightforward than many previouslyreported studies. Firstly, the source object in our study (the TEEprobe) is always the same. Secondly, the visibility and contrast ofthe TEE probe in the X-ray images is relatively constant and is un-likely to be affected by other objects because the density of the TEE

Fig. 13. TEE volumes were registered with and overlaid onto the X-ray images fortwo patients undergoing EP procedures. The original X-ray images are shown in thefirst row. For the first patient (first column), the TEE volume shows a deca-polarcatheter positioned in right atrium. The dots highlight the position of the deca-polarcatheter in the background X-ray image. For the second patient (second column), alasso catheter was inserted into the left atrium. For both patient data, theregistration was successful as the catheters in the TEE volumes showed a goodalignment with the background X-ray images.

Table 3The average computation time of the 2D–3D registration.

Num. of X-rayimages

Image size Num. ofregistrations

Mean time(s)

SD(s)

1 1024 � 1024 8400 8.0 2.12 1024 � 1024 4200 11.3 4.11 512 � 512 12,800 1.8 0.62 512 � 512 6400 5.4 1.6

G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 47

probe is considerably higher than the soft tissues and the spine.The robustness study described in Section 3 showed consistentregistration results when the start positions were within the cap-ture range of the 2D–3D registration algorithm. Our study showedthe capture range of the 2D–3D registration was approximately10 mm in terms of mTRE. An additional examination of all thefailed registrations found that most of the failed registrationsstarted outside the region of 10� � 10 mm in terms of the magni-tudes of rotations and translations, suggesting that an initial man-ual alignment was required to bring the nano-CT volume into the

Fig. 14. Co-registered echo and X-ray images for one of the TAVI patients. (a) The X-ray imreplacement valve; (b) the 3D TEE reveals cardiac anatomy including the aorta and the vathe replacement valve; (c) the highlighted area in the combined image of TEE and X-rventricle.

capture range before the automated registration would be success-ful. However, the end point of the proposed method is to registerthe TEE images with the X-ray images.

The accuracy of our TEE probe calibration did not comparefavorably with the sub-millimeter results obtained from classicallytracked ultrasound probes (Mercier et al., 2005). We repeated ourcalibration to check for any mistakes that may have been madeduring the calibration process but the results were very similarto our initial experiment. Looking at Eq. (3), we see that thereare three sources of error. Firstly, there is the error of locating fidu-cials in the US image space; secondly, there is the error of locatingfiducials in the 2D X-ray image space and back-projecting these to3D C-arm space; and finally, there is the error of performing the2D–3D registration. For classically tracked US probes, the trackingerror will be in the order of 1–2 mm, in the case of EM tracking, orless, in the case of optical tracking. For tracking using 2D–3D reg-istration, we envisage the errors to be greater and furthermore wemust include the error of the back-projection, which is typically1 mm. Therefore, the reported results of a 2 mm residual errorfrom Eq. (3) and 3D mTRE of 4.6/5 mm are expected from the cur-rent methodology. There could be scope to improve this substan-tially using a multi-crosswire phantom coupled to a largenumber of ultrasound and X-ray image acquisitions for calibration.

The X-ray and TEE image registration results of both the phan-tom and clinical studies are encouraging in terms of the target reg-istration accuracy achieved. The clinical accuracy requirement isinfluenced by the type of clinical procedure (Linte et al., 2010).The values of less than 3 mm achieved for the heart phantom and1.5–4 mm achieved for the clinical cases are within the accuracyrequirement for navigating within the cardiac chambers and greatvessels that have a minimum diameter of approximately 10 mmin adults. However, in this study, as for many other clinical studies,it was difficult to find a gold standard. mPD can be used as an indi-cator for the accuracy but for the TAVI cases, the interventional

age shows the interventional devices including the catheter, the guide wire and thelves as well as the catheter. However, it was difficult to identify the exact location ofay shows the replacement valve had passed through the aortic valve into the left

Table 4The point-to-surface error represents the error in the registration of TEE and MRI in C-arm space. For LV and RV, the points were defined on all the six TEE volumes, 10 foreach volume. For LA, RA and the valves, points were defined on the volumes wherethe borders were clearly visible. The valves were not segmented from the MR volume.The point-to-surface distance for the valves was the distance of a point defined on thevalves to the top surface of the LV or the RV.

Num. of points Mean error (mm) SD (mm) Max. error (mm)

LV 66 2.0 1.5 4.7RV 60 1.9 1.2 4.9Valves 16 1.9 1.3 2.7LA 22 1.5 0.8 2.8RA 16 2.0 1.4 4.4LVPM 20 2.4 1.8 3.2Overall 200 1.9 1.2 4.9

Table 5Registration error results from the clinical cases. Devices were projected from the TEEvolume data to the X-ray image data using the computed 2D–3D registration. ThemPD was computed for each device.

Patient ID Procedure mPD (mm) SD (mm)

1 EP 3.1 2.62 EP 1.5 1.62 EP 3.2 2.53 TAVI 2.7 1.84 TAVI 4.2 3.05 TAVI 2.3 2.1

48 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

devices are relatively straight in the FOV of the TEE images andtherefore the tangential error along the interventional devices can-not be captured. However, the validity of mPD as an accuracy met-ric was supported in the unsuccessful TAVI case, where the clinicaloutcome matches the finding of the TEE and X-ray registration. Theaccuracy of the proposed registration method is also confirmed inone of the EP cases where a circular catheter was used as the refer-ence for mPD calculation.

At this stage, all the clinical data we used were processed off-line. The X-ray images and the TEE volumes were not acquired atthe same time. Although we attempted to minimize the registra-tion error by using the images acquired in the same cardiac andrespiratory phases, the error caused in the cardiac phase estima-tion will be propagated to the end registration result. Such erroris likely to be further reduced in the future when real-time X-rayand ultrasound data can be acquired simultaneously.

The jittering motion of the TEE probe during a clinical procedureis another source of error which potentially could affect the accu-racy of the TEE and X-ray registration. Experimental results showthe error introduced by the jittering motion can be as much as a4 mm error in the probe localization. Fig. 8a suggests this errorcan be easily removed using the 2D–3D registration. The successrate of the 2D–3D image registration is 100% given the starting po-sition of the source volume is within 5 mm capture range. However,the speed of the current 2D–3D registration algorithm needs to beimproved considerably to satisfy the requirement of real-timetracking.

5.2. Comparison with EM tracking systems

At present, our 2D–3D registration algorithm takes around 5 s todetermine the 3D position of the TEE probe using two 512 � 512 X-ray images. EM tracking systems determine the location of objectsthat are embedded with the EM sensors in real-time. More impor-tantly, it is not required that the EM sensor be in the line-of-sightof the magnetic field generator, as for optical tracking systems. Pre-vious studies (Gao et al., 2009; Jain et al., 2009) confirmed that itwas possible to localize the TEE probe by using an EM tracking sys-

tem, although neither of the two studies were performed in the clin-ical environment. At present, there is no commercial EM tracker-embedded TEE probe available. To use an EM tracking system withthe TEE probe, the TEE probe must be modified so that an EM sensorcan be fitted within it. Compared to the use of an EM tracker, the firstadvantage of our method is that it does not use any additional track-ing device. The accuracy of the EM tracking system relies on a staticmetal environment. With the presence of the moving X-ray C-armand other medical devices including the TEE probe itself, the reli-ability of the EM tracking system is questionable (Hastenteufel etal., 2006). Additionally, the methodology presented for the co-regis-tration of 3D TEE and X-ray fluoroscopy data is better suited for theroutine clinical work flow.

5.3. Mono-plane X-ray and bi-plane X-ray

The 2D–3D registration algorithm is not sensitive to detectingmisalignment along the direction of the X-ray beam so that the2D–3D registration algorithm using mono-plane images oftenproduces considerable error in the direction of projection.Introducing one more X-ray image acquired from a different angleeffectively solves this problem, allowing an accurate registration inall six DOFs although the radiation dose will be double (but verysmall when compared to entire procedure dose). This can beperformed using bi-plane X-ray systems but these are not com-monly available. A more conventional method is to use sequentialbi-plane images from a mono-plane X-ray system. However, withsequential bi-plane acquisition, correct phase-matching (for bothrespiratory phase and cardiac phase) of the X-ray data must becarried out.

Clinically, the 2D–3D registration with bi-plane X-ray is not al-ways necessary. The clinical workflow includes two different waysto visualize and utilize the registration results, Echo ? X-ray pro-jection and X-ray ? Echo projection. Echo ? X-ray projectionoverlays the echo volume on top of the X-ray images producing a‘roadmap’ to navigate the clinical procedures such as atrial fibrilla-tion (AF) ablation and TAVI. In these cases, the 2D–3D registrationusing mono-plane X-ray is sufficient as the accuracy in the direc-tion of projection is not a major concern. However, in cases whichthe 3D position of the interventional devices must be preciselylocalized, bi-plane X-ray images will be needed to ensure a preciseTEE and X-ray registration in all six degrees of freedom. For exam-ple, in some AF ablation cases, the locations of the ablation sitesneed to be recorded for reviewing purposes. An example for theapplication of X-ray ? Echo projection could be: (1) determinethe location of the ablation catheter tip from the X-ray image;(2) back-project the 2D location of the catheter tip to C-arm space,producing a line intersecting the registered TEE volume; and (3)determine the 3D position of the ablation catheter tip either man-ually or automatically from the TEE volume.

5.4. Limitations and future work

One of the main objectives of this study was to assess the clinicalfeasibility of the proposed method. Although all the clinical datawere acquired specifically for this study, the data processing wasdone off-line. However, the transition to real-time or near-real-time capability is relatively straightforward in this case, with allthe required components already present, including live datastreaming from both X-ray and ultrasound systems. Our futurework will focus on the implementation of the live functionalityand also improvements in the visualization of the co-registereddata. The 3D TEE images were acquired in full volume mode. OurTEE probe was calibrated using full volume echo images. The clini-cal procedures are usually guided by live 3D echo images whoseFOV is about a quarter of the full volume images. Therefore,

G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 49

co-registration with a global roadmap from either CT or MR imagedata could assist in clarifying the anatomical context of the smallFOV echo data.

6. Conclusions

In this study, we have described a novel TEE to X-ray fluoros-copy registration technique. The method was successfully evalu-ated using phantom data and five clinical datasets acquired fromtwo different types of cardiac interventional procedures, includingcardiac EP procedures and TAVI procedures. The experimental re-sults show that our method is fast, robust and accurate. It is likelythat such a co-registration and visualization technology is going tohave a significant impact in the field of image-guided cardiacinterventions.

References

Barker, G.H., Shirali, G., Ringewald, J.M., Hsia, T.Y., Bandlsode, V., 2008. Usefulness oflive three-dimensional transesophageal echocardiography in a congenital heartdisease center. The American Journal of Cardiology 103, 1025–1028.

Brost, A., Liao, R., Hornegger, J., and Strobel, N., 2010. 3-D Respiratory MotionCompensation during EP Procedures by Image-Based 3-D Lasso Catheter ModelGeneration and Tracking. Medical Image Computing and Computer AssistedIntervention (MICCAI’09) Medical Image Computing and Computer AssistedIntervention (MICCAI’09), pp. 394–401.

De Buck, S., Meas, F., Ector, J., Bogaert, J., Dymarkowski, S., Heidbuchel, H., Suetens,P., 2005. An augmented reality system for patient-specific guidance of cardiacablation procedures. IEEE Transactions on Medical Imaging 24, 1512–1524.

Gao, G., Penney, G., Gogin, N., Cathier, P., Arujuna, A., Wright, M., Caulfield, D.,Rinaldi, A., Razavi R., Rhode, K.S., 2010. Rapid Image Registration of Three-dimensional Transesophageal Echocardiography and X-ray Fluoroscopy for theGuidance of Cardiac Interventions. Information Processing in Computer-Assisted Intervention (IPCAI’10) Information Processing in Computer-AssistedIntervention (IPCAI’10).

Gao, G., Reddy, K., Ma, Y., Rhode, K.S., 2009. Real-time Compounding of Three-Dimensional Transesophageal Echocardiographic Volumes: The Phantom Study.IEEE Engineering in Medicine and Biology Society (EMBC’09) IEEE Engineeringin Medicine and Biology Society (EMBC’09), pp. 499–502.

Hastenteufel, M., Vetter, M., Meinzer, H.P., Wolf, I., 2006. Effect of 3D ultrasoundprobes on the accuracy of electromagnetic tracking systems. Ultrasound inMedicine and Biology 32, 1359–1368.

Hawkes, D.J., Mol, C.B., Colchester, A.C.F., 1987. The accurate 3-D reconstruction ofthe geometric configuration of vascular trees from X-ray recordings. Physicsand Engineering of Medical Imaging 250, 258.

Hipwell, J.H., Penney, G., Mclaughlin, R.A., Rhode, K.S., Summers, P., Cox, T.C., Byrne,J.V., Noble, J.A., Hawkes, D.J., 2003. Intensity based 2D–3D registration ofcerebral angiograms. IEEE Transactions on Medical Imaging 22, 1417–1426.

Huang, X.S., Ren, J., Guiraudon, G., Boughner, D., Peters, T., 2009. Rapid dynamicimage registration of the beating heart for diagnosis and surgical navigator. IEEETransactions on Medical Imaging 28, 1802–1814.

Jain, A., Gutierrez, L., Stanton, D., 2009. 3D TEE Registration with X-ray Fluoroscopyfor Interventional Cardiac Applications. Functional Imaging and Modeling of theHeart (FIMH’09) Functional Imaging and Modeling of the Heart (FIMH’09), pp.321–329.

King, A.P., Boubertakh, R., Rhode, K.S., Ma, Y., Chinchapatnam, P., Gao, G., Tangcharoen,T., Ginks, M., Cooklin, M., Gill, J.S., Hawkes, D.J., Razavi, R., Schaeffter, T., 2009. Asubject-specific technique for respiratory motion correction in image-guidedcardiac catheterization procedures. Medical Image Analysis 13, 419–431.

King, A.P., Rhode, K.S., Ma, Y., Yao, C., Jansen, C., Razavi, R., Penney, G., 2010.Registering preprocedure volumetric images with intraprocedure 3-Dultrasound using an ultrasound imaging model. IEEE Transactions on MedicalImaging 29, 924–937.

Klemm, H., Steven, D., Johnsen, C., Ventura, R., Rostock, T., Lutomsky, B., Risius, T.,Meinertz, T., Willems, S., 2007. Catheter motion during atrial ablation due to thebeating heart and respiration: impact on accuracy and spatial referencing inthree-dimensional mapping. Heart Rhythm 4, 587–592.

Knecht, S., Wright, M., Akrivakis, S., Nault, I., Matsuo, S., Chaudhry, G.M., Haffajee, C.,Sacher, F., Lellouche, N., Miyazaki, S., Forclaz, A., Jadidi, A.S., Hocini, M., Ritter, P.,

Clementy, Haissaguerre, M., Orlov, M., Jais, P., 2010. Prospective randomizedcomparison between the conventional electroanatomical system and three-dimensional rotational angiography during catheter ablation for atrialfibrillation. Heart Rhythm 7, 459–465.

Kovoor, P., Ricciardello, M., Collins, L., Uther, J., Ross, D., 1998. Risk to patients fromradiation associated with radiofrequency ablation for supraventriculartachycardia. Circulation 98, 1534–1540.

Linte, C.A., Moore, J., Peters, T.M., 2010. How Accurate is Accurate Enough? A BriefOverview on Accuracy Considerations in Image-Guided Cardiac Interventions.In: Proceedings of 32nd Annual International Conference of the IEEE EMBS, pp.2313–2316.

Mackensen, G.B., Hegland, D., Rivera, D., Adams, D.B., Bahnson, T.D., 2008. Real-time3-dimensional transesophageal echocardiography during left atrial radiofrequency catheter ablation for atrial fibrillation. Circulation: CardiovascularImaging 1, 85–86.

Manzke, R., Meyer, C., Ecabert, O., Peters, J., Noordhoek, N., Thiagalingam, A., Reddy,V., Chan, C.C., Weese, J., 2010. Automatic segmentation of rotational X-rayimages for anatomic intra-procedural surface generation in atrial fibrillationablation procedures. IEEE Transactions on Medical Imaging 29, 260–272.

Mercier, L., Langø, T., Lindseth, F., Collins, D.L., 2005. A review of calibrationtechniques for freehand 3-D ultrasound systems. Ultrasound in Medicine &Biology 31, 449–471.

Modan, B., Keinan, L., Blumsterin, T., Sadetzki, S., 2000. Cancer following cardiaccatheterization in childhood. International Journal of Epidemiology 29, 424–428.

Penney, G., Blackalll, J.M., Hamady, M.S., Sabharwal, T., Adam, A., Hawkes, D.J., 2004.Registration of freehand 3D ultrasound and magnetic resonance liver images.Medical Image Analysis 8, 81–91.

Penney, G., Weese, J., Little, J.A., Desmedt, P., Hill, D.L., Hawkes, D.J., 1998. Acomparison of similarity measures for use in 2-D–3-D medical imageregistration. IEEE Transactions on Medical Imaging 17, 586–595.

Razavi, R., Hill, D.L., Keevil, S.F., Miquel, M.E., Muthurangu, V., Hegde, S., Rhode, K.S.,Barnett, M., van Vaals, J., Hawkes, D.J., Barker, E., 2003. Cardiac catheterizationguided by MRI in children and adults with congenital heart disease. The Lancet362, 1877–1882.

Rhode, K.S., Hill, D.L., Edwards, P.J., Hipwell, J.H., Rueckert, D., Sanchez-Ortiz, G.,Hegde, S., Rahunathan, V., Razavi, R., 2003. Registration and tracking tointegrate X-ray and MR images in an XMR facility. IEEE Transaction onMedical Imaging 22, 1369–1378.

Rhode, K.S., Sermesant, M., Brogan, D., Hegde, S., Hipwell, J.H., Lambiase, P.,Rosenthal, E., Bucknall, C., Qureshi, S.A., Gill, J.S., Razavi, R., Hill, D.L., 2005. Asystem for real-time XMR guided cardiovascular intervention. IEEE Transactionon Medical Imaging 24, 1428–1440.

Roche, A., Pennec, X., Malandain, G., Ayache, N., 2001. Rigid registration of freehand3D ultrasound with MR images: a new approach combing intensity andgradient information. IEEE Transactions on Medical Imaging 20, 1038–1049.

Shekhar, R., Zagrodsky, V., 2002. Mutual information-based rigid and non-rigidregistration of ultrasound volumes. IEEE Transactions on Medical Imaging 21,9–22.

Silverstry, F.E., Kerber, R.E., Brook, M.M., Caroll, J.D., Eberman, K.M., Goldstein, S.A.,Herrmann, H.C., Homma, S., Mehran, R., Packer, D.L., Parisi, A.F., Pulerwitz, T.,Seward, J.B., Tsang, T.S., Wood, M.A., 2009. Echocardiography-guidedinterventions. Journal of American Society of Echocardiography 22, 213–231.

Sra, J., Narayan, G., Krum, D., Malloy, A., Cooley, R., Bhatia, A., Dhala, A., Blanck, Z.,Nangia, V., Akhtar, M., 2007. Computed tomography–fluoroscopy imageintegration-guided catheter ablation of atrial fibrillation. Electrophysiology18, 409–414.

Turgeon, G.-A., Lehman, G., Guiraudon, G., Drangova, M., Holdsworth, D., Peters, T.,2005. 2D–3D registration of coronary angiograms for cardiac procedureplanning and guidance. Medical Physics 32, 3737–3749.

van de Kraats, E., Penney, G., Tomazevic, D., van Walsum, T., Niessen, W., 2005.Standardized evaluation methodology for 2-D–3-D registration. IEEETransactions on Medical Imaging 24, 1177–1189.

Wein, W., Carnus, E., John, M., Diallo, M., Duong, C., Al-Ahmad, A., Fahrig, R.,Khamene, A., Xu, C., 2009. Towards guidance of electrophysiological procedureswith real-time 3D intracardiac echocardiography fusion to C-arm CT. MedicalImage Computing and Computer Assisted Intervention (MICCAI’09) MedicalImage Computing and Computer Assisted Intervention (MICCAI’09), pp. 9–16.

Yu, H., Fahrig, R., Pelc, N.J., 2005. Co-registration of X-ray and MR fields of view in ahybrid XMR system. Journal of Magnetic Resonance Imaging 22, 291–301.

Yushkevich, P., Piven, J., Hazlett, H., Smith, R., Ho, S., Gee, J., Gerig, G., 2006. User-guided 3D active contour segmentation of anatomical structures: significantlyimproved efficiency and reliability. Neuroimage 31, 1116–1128.