Automatic 3D-2D image registration using partial digitally reconstructed radiographs along projected anatomic contours

Xin Chen1, Martin R. Varley1, Lik-Kwan Shark1, Glyn S. Shentall2, Mike C. Kirby2

1ADSIP Research Centre, University of Central Lancashire, Preston, U.K. 2Rosemere Cancer Centre, Royal Preston Hospital, U.K.

{xchen2, mrvarley, lshark @uclan.ac.uk; glyn.shentall, mike.kirby @lthtr.nhs.uk}

Abstract

The paper presents a new intensity-based 3D-2D image registration algorithm for automatic pre-treatment validation in radiotherapy. The novel aspects of the algorithm includes a hybrid cost function developed based on partial digitally reconstructed radiographs (DRRs) generated along projected anatomic contours and level set for similarity measurement, and a fast search method developed based on parabola fitting and sensitivity based search order. Using CT and orthogonal X-ray images from a skull phantom, the proposed algorithm is compared with the conventional ray-casting full DRR based registration method. Not only is the algorithm shown to be computationally more efficient with registration time being reduced by a factor of 8, but also the algorithm is shown to offer 50% higher capture range allowing the initial patient displacement up to 15 mm (measured by mean target registration error) and high registration accuracy with average errors of 0.53±0.12 mm for translation and 0.61˚±0.29˚ for rotation within the capture range.

1. Introduction

External beam radiotherapy is a complex treatment that requires a number of different processes in order to give a high dose of high energy radiation to kill cancerous cells. These include: (1) volume imaging (such as Computerised Tomography) and treatment planning for deciding where and how to deliver the course of radiotherapy; (2) pre-treatment checks and verification (to ensure that the patient’s treatment will be accurate), (3) treatment delivery (with machines for producing high energy radiation beams), and (4) on-treatment verification (for ensuring that the high dose of radiation is given correctly and in the right place throughout the treatment course). The processes in stages 1 and 2 above can take from a few days to a

couple of weeks. Any steps which may safely be taken to make them more efficient are of great value to the patient. At present, for most pre-treatment validation systems, two orthogonal kilo-volts X-ray images (anterior-posterior and lateral views) taken from the simulator are compared with the corresponding digitally reconstructed radiographs (DRRs), derived from the planning CT data, to determine the final machine and patient set-up for treatment delivery. Clinicians are required to identify and match manually some rigid bony features from the kilo-volts X-ray image pair (known as the simulator images) and the corresponding DRR image pair in order to obtain the data for position modification in three dimensions. The main problems with this approach are (1) low resolution of the DRR images giving rise to errors in visual comparison; and (2) only translation errors are considered without taking out-of-plane rotations into account. In addition, the matching accuracy depends highly on the clinician’s experience which may directly affect the quality of the final treatment.

Among the automatic methods proposed for 3D-2D registration, there are mainly two different approaches. The first approach is based on image features [1] and requires feature extraction from both 2D X-ray and 3D CT images. Once the feature extraction is done, the registration process can be very fast, and the registration accuracy depends mainly on the feature extraction accuracy. The second approach is based on pixel and voxel intensity [2] and requires DRRs to be generated from the 3D CT data. The unknown pose of the CT volume is estimated by optimising the similarity measurement between DRRs and 2D X-ray images. Although it has shown to be an accurate and robust registration method, the efficiency and accuracy of the system depends highly on the computation time and quality of the DRRs, as well as the performance of the optimisation procedure. For the proposed method, 3D treatment planning CT data and two orthogonal kilo-volts X-ray images

Medical Information Visualisation - BioMedical Visualisation (MediViz 2007)0-7695-2904-6/07 $25.00 © 2007

are used to achieve 3D-2D registration, which combines the idea of the level-set method [4] and partial DRR generation to achieve fast and accurate 3D-2D registration. The paper is organised into four sections. In Section 2, a detailed description of the proposed 3D-2D registration framework is presented. Evaluation methods and experiment results are given in Section 3. Conclusions are drawn in Section 4.

2. Methodology

The proposed algorithm takes the advantages of the intensity-based method, but with a more efficient way of generating DRRs. This is achieved through an iterative process that aligns automatically a pair of orthogonal 2D X-ray images of the current patient pose with the corresponding 3D CT image of the patient acquired at the treatment planning stage. As illustrated in Figure 1 using a skull as an example, the proposed algorithm consists of the following 6 main steps:

(1) 3D mesh model construction: a 3D mesh model is constructed via threshold based extraction of the bony structures from the volumetric planning CT data of the patient.

(2) 3D-2D projection: For a given patient pose, with the initial pose (planned patient set-up) at the start and estimated new pose from step 6, two orthogonal silhouettes are generated by projection of the 3D mesh model from 3D to 2D, and this is followed by extraction of the contours along the silhouettes to yield the corresponding 2D anatomic contours.

(3) Partial X-ray image extraction: By projecting the anatomic contours to the X-ray image plane, a short X-ray image segment is extracted from the X-ray image at each down sampled point along the projected contour.

(4) Partial DRR generation: A set of DRR segments are generated from 3D CT data from the same positions where the X-ray segments are extracted. It results in the dominant anatomic features being contained in a DRR strip for each of the orthogonal 2D projections, and the normal full DRR generation is no longer needed leading to significant reduction in computation time.

(5) Similarity measurement: The similarity between the DRR and X-ray strips is measured based on mutual information. To ensure rapid convergence to the optimum alignment position and a wider convergence range, a hybrid cost function is proposed, which uses not only mutual information but also a level-set term [4] based on the statistical properties of X-ray images.

(6) Optimisation: The best neighbour search optimisation method combined with parabolic fitting is used to generate a better pose for iteration from step 2 by estimating a better geometric transform parameter value.

Figure 1 Overview of proposed algorithm

2.1. Mesh model construction and 3D to 2D projection

In order to match the 3D CT data of a patient with the corresponding 2D X-ray images, the exact projection geometry has to be known accurately. The method of estimating the projection geometry parameters can be found in [1]. With the position of the X-ray source fixed, different poses of 3D CT data can be produced by geometric transformation of the CT volume coordinates using the composite transformation matrix T given by

−−++−

=

1000z

y

x

TcccssTcscssccsssscTsscsccsssccc

Tαββαβ

γααβγαγαβγγβ

γααβγαγαβγγβ

(1) where c and s denote cos and sin functions, subscripts α, β and γ denote the rotation angles alpha, beta and gamma around the three axes and Tx, Ty and Tz are the translations in the three directions. The geometrical significance of these parameters is illustrated in Figure 2. In the proposed algorithm, a mesh model (Figure 3(a)) was first constructed from the planning CT data by setting an appropriate threshold for identifying bones and applying the marching cube algorithm [5]. With the projection geometry based on the planned patient set-up at the start and estimated geometric transformation parameters in subsequent iterations, an orthogonal pair of silhouettes projected from the mesh model (Figure 3 (b)) can be generated. This is followed

(1) 3D mesh model construction

(2) 3D-2D projection

(5) Similarity measurement

(4) Partial DRR generation

Planning CT data

(6) Optimisation(3) Partial X-ray

extraction

Minimum? No

Transformation parameters

Yes

Orthogonal X-ray image pair

Medical Information Visualisation - BioMedical Visualisation (MediViz 2007)0-7695-2904-6/07 $25.00 © 2007

by extraction of the contours along the silhouettes to yield the required anatomic contour in 2D.

Figure 2 Coordinate of volume data

(a) (b)

Figure 3 (a) 3D mesh model constructed from planning CT data (b) Orthogonal pair of 2D projections from mesh model

2.2. X-ray image extraction and partial DRR generation

The computation of DRRs is based on volume rendering techniques. In the literature, there are many conventional and accelerated DRR generation methods, such as ray casting [6], shear-warp factorization [2] and pre-computed attenuation fields [3]. The ray casting method was used for the proposed algorithm because it produces the best image quality and the computation time can be improved significantly by using advanced CPUs or using graphic card programming. More importantly, the proposed algorithm only requires a set of small partial DRRs to be generated at specific regions instead of generating full DRRs. In this case, the shear-warp method does not significantly improve the computation time, and it is not efficient to pre-compute attenuation fields for partial DRR generation. Since the regions surrounding the anatomic contours are seen to contain more useful information and provide distinctive anatomic structures for registration, a series of small partial DRRs along the projected anatomic contours is generated for matching rather than generation of the full size DRRs. With the projected contours on the X-ray image plane obtained in Step 2, the points on the contours are first down sampled in this step, and a short X-ray

image segment is extracted across the contours for each sample point. As shown in Figure 4 (a), the black lines represent the projected contours from the 3D mesh model, whereas the white segments with the same length are the positions for extracting partial X-ray images. Figure 4 (b) shows the orthogonal X-ray strips. Based on the extracted regions in the orthogonal X-ray images, the corresponding partial DRRs are generated using the ray-casting method. Figures 4 (c) and 4 (d) show the resulting partial DRR contours and the partial DRR strips, respectively. These two sets of corresponding strips (Figures 4 (b) and 4 (d)) will be used in the similarity measurement in the next step. The length of the segment is determined experimentally based on the original size of the input X-ray image. For an X-ray image with 512 by 512 pixels, the segment length is set to 31 pixels.

(a)

(b)

(c)

(d)

Figure 4 (a)/ (c) Orthogonal X-ray/ DRR images with regions highlighted for X-ray/ DRR strip extraction/ generation; (b)/ (d) orthogonal X-ray /DRR strips

2.3. Similarity measurement

There are various similarity measurement methods available for intensity based registration, such as normalised correlation coefficient, gradient correlation, mutual information, and pattern intensity. The detailed information about these similarity measurement methods can be found in [7]. Since the CT scan and X-ray images were taken at different effective energies, 80keV and 50-60keV,

Medical Information Visualisation - BioMedical Visualisation (MediViz 2007)0-7695-2904-6/07 $25.00 © 2007

respectively, it is difficult to generate DRRs exactly like the X-ray images. Hence, the normalised mutual information (NMI) is used to measure the similarity between these two different image modalities. The behaviour of the NMI similarity measurement based on partial DRRs was evaluated by varying each transformation parameter, Tx, Ty, Tz, α, β and γ around the true value one at a time and keeping the others with the true values. The cost function is shown to be good in the sense that the minimum is distinct at the correct location (see curves in Figure 6 with weighting parameter w=0). However, it is not particularly smooth for rotation parameters which might introduce problems in the optimisation process. In order to make the cost function smoother and more stable, a second term is introduced to the cost function based on the idea of the level set method. By superimposing the projected anatomic contours obtained in step 2 with the corresponding X-ray image, each X-ray image is divided into two regions – the region inside the contour and region outside (see Figure 5). This enables a cost function term to be built based on some statistical properties of the two regions, resulting in the function value to reach minimum when the contours are perfect aligned with the object. Similar to [4], the cost function term used in the proposed algorithm is given by

)(2max

22

outin

outinLS AAI

AAE+

+−= νµ (2)

where Ain and Aout represent the areas inside and

outside the contour respectively, and µ is the average intensity inside the contour and ν is the average intensity outside the contour. The denominator is a normalisation term, with Imax denoting the maximum value of the X-ray image. The cost function term has shown to be smooth in parameter space, but the minimum values for some of the parameters are located at a wrong location.

Figure 5 X-ray image is divided into two regions by the contours: (a) region inside the contour (b) region outside the contour

The proposed algorithm then combines the above two terms to produce one, which maintains smoothness and accuracy properties, expressed as:

LSNMITot wEEE += (3) where ENMI is the inverted NMI measurement between the orthogonal DRR strips and their corresponding X-ray strips (producing a minimum for the best match), ELS is the level set term, and w is the weighting parameter to balance the effect of the two terms. Figure

Figure 6 Proposed cost functions for translations and rotations with different weighting parameter

Medical Information Visualisation - BioMedical Visualisation (MediViz 2007)0-7695-2904-6/07 $25.00 © 2007

6 shows the behaviour of the proposed hybrid cost function with different weightings. It is observed that the advantages offered by both terms are successfully combined. Not only does it keep a sharp peak at the correct location produced by the NMI term, but also it maintains the smoothness produced by the level set term. From Figure 6, the weighting parameter w was chosen to be 30 for implementation, since it enables a fast convergence and a wider convergence range.

2.4. Optimisation

The best neighbour search optimisation method combined with a parabolic interpolation method was developed for step 6 to speed up the search of the optimum pose. The search of six rigid transformation parameters is divided into individual one-dimensional search. For each parameter the cost function is evaluated three times at the current position and its negative and positive neighbours, respectively, with the initial distance between the current position and its neighbours pre-defined by a searching range. A parabola is then fitted to the computed three cost function values to enable selection of the best transformation parameter value for the next iteration. There are three selection criteria: (1) A minimum is found by calculating the first derivative of the fitted parabola with the second derivative being positive: In this case, the minimum is selected as the current best transformation parameter value for evaluating the next parameter or the next iteration; (2) A maximum is found by calculating the first derivative of the fitted parabola with the second derivative being negative: In this case, the transformation parameter value corresponding to the smallest cost function value is selected; (3) A minimum is found, but it is too far away from the evaluated position (due to the cost function being too flat): In this case, the transformation parameter value corresponding to the smallest cost function value is again selected. When a better position was found for one transformation parameter, it will be used for evaluating the next one. When all the transformation parameters satisfy the first criterion, the searching range is divided by a factor to refine the transformation estimation results. The whole optimisation will be terminated when the searching range is smaller than a pre-set threshold.

By parabola fitting for parameter search rather than tracing down the hill to find a definite minimum, it not only saves computation time, but also prevents the pose from being dragged to some incorrect minimum in the searching space.

To speed up further the optimisation process, the order of transformation parameter search is arranged according to the sensitivity of different parameters to different views. Through the experiments (and see Figure 3), it is observed that the translation parameter Tx and rotation parameter β are only sensitive to the changes in anterior-posterior (AP) view X-ray image, whereas Ty and α are sensitive to the changes in the right lateral (RL) view X-ray image. Both views are needed to estimate the parameters Tz and γ. Hence, the reduction of two view DRR generation to one view, for some of the parameters, leads to a more computation efficient algorithm without sacrificing the registration accuracy.

3. Performance evaluation

The proposed method has been evaluated using planning CT data and X-ray images from a skull phantom. A widely used 3-D error measurement for performance evaluation is the mean target registration error (mTRE) [8] defined as:

∑=

−=k

iigoldireggoldreg pTpT

kTTPmTRE

1, ||||1),(

(4) where P contains a set of 3D points defined in the experiments by the vertices in the constructed 3D mesh model with k denoting the number of vertices, Treg and Tgold are the measured transformation matrix and the true transformation matrix, respectively, with each containing all six transformation parameters. For performance evaluation, the simulated orthogonal X-ray image pairs were generated with the initial displacement set by random variation of the six transformation parameters. Based on the mTRE, the initial displacements are grouped into thirteen intervals from 0-2 mm to 24-26 mm with 30 pairs of simulated orthogonal X-ray image pairs per interval and 512 by 512 pixels per image. In the registration process, a two level multi-resolution optimisation was used to speed up the computation, the original image was smoothed and down-sampled with a factor of 4 in the first stage and the original image was used in the second stage. Although the sub-sampled image may cause a sparse joint histogram, it did not affect the registration results in the experiments. Whilst the initial and final searching ranges for the first optimisation stage is 5 mm and 0.5 mm, respectively; they are set to 1 mm and 0.1 mm, respectively, for the second stage. Figure 7 shows a set of registration results, where coordinates on the horizontal axis correspond to the initial patient displacement before registration, and coordinates on the vertical axis correspond to the

Medical Information Visualisation - BioMedical Visualisation (MediViz 2007)0-7695-2904-6/07 $25.00 © 2007

registration error (for translation in Figure 7a and for rotation in Figure 7b) after applying the proposed algorithm. The two curves show the variation of root mean squared registration error (rms) against the initial patient displacement for all 390 data pairs, with vertical high-low bars superimposed on the curve showing the range of the standard deviation (std). From Figure 7, it is seen that, for the initial patient displacement up to 15 mm, the rms translation and rotation errors including standard deviation after registration is within 1 mm and 1˚, respectively, both within the defined clinical useful levels. Compared with the conventional method based on ray-casting full-size DRR generation, the proposed algorithm increases the capture range of the initial patient displacement by 50%, since the conventional method was found to be able to provide the clinically required rotation accuracy for the initial patient displacement upto 10 mm from the tests performed.

Figure 7 Registration errors for (a) translation, and (b) rotation

4. Conclusions

This paper has presented a novel 3D-2D image registration algorithm for pre-treatment validation in radiotherapy. It offers a high capture range allowing an

initial patient displacement up to 15 mm (measured by mTRE), and high registration accuracy with average errors of 0.53±0.12 mm for translation and 0.61˚±0.29˚ for rotation (rms ± std) within the capture range. Furthermore, the computation time of the proposed algorithm is about 120s running in a 2.4 GHz P4 PC in an un-optimised environment, which is 8 times less than the conventional full-size DRR registration based on the ray-casting method. It is noteworthy that great advances are currently being made in image guided radiotherapy. These advances may remove the need for pre-treatment check entirely by enabling better and more accurate imaging during treatment, but at a cost of more radiation to healthy parts of the body. The work presented in this paper is also applicable with these newer technologies, enabling them to be used more frequently throughout the course of the therapy, but with lower radiation doses to healthy tissue.

References

[1] X. Chen, M. R. Varley, G. S. Shentall, M. C. Kirby and L. –K. Shark. An extension of iterative closest point algorithm for automatic pre-treatment validation in radiotherapy. IEEE 3rd International Conference on Biommedical Visualisation, MediViz06, London, July 2006. [2] P. Lacroute and M. Levoy. Fast volume rendering using a shear-warp factorisation of the viewing transformation. Proceedings of the 21st annual conference on computer-graphics and interactive techniques (SIG-GRAPH 94) (ACM, New York, 1994). Pp. 451-458, 1994. [3] D. B. Russakoff, T. Rohlging, K. Mori, D. Rueckert, A. Ho, J. R. Adler, Jr. and C. R. Mauer, Jr. Fast generation of digitally reconstructed radiographs using attenuation fields with application to 2D-3D image registration. IEEE Trans. Med. Imag. 24(11), 1441-1454, 2005. [4] A. Tsai, A. Yezzi Jr., W. Wells, C. Tempany, D. Tucker, A. Fan, W. E. Grimson and A. Willsky. A shape-based approach to the segmentation of medical imagery using level set. IEEE Trans. Med. Imag. 22(2), 2003. [5] W. Lorensen and H. E. Cline. Marching cubes: A high resolution 3-d surface construction algorithm. Proc. SIGGRAPH:Computer Graphics, 21, 1987. [6] G. W. Sherouse, K. Novins and E. L. Chaney. Computation of digitally reconstructed radiographs for use in radiotherapy treatment design. Int. J. Radiation Oncology Biology Physics, 18(3), pp. 651-658, 1990. [7] G. P. Penney, J. Weese, J. A. Little, P. Demedt, D. L. G. Hill and D. J. Hawkes. A comparison of similarity measures for use in 2-D – 3-D medical image registration. IEEE Trans. Med. Imag. 17(4), 1998. [8] E. B. Van de Kraats, G. P. Penney, D. Tomazevic, T. Van Walsum and W. J. Niessen. Standardized evaluation methodology for 2-D—3-D registration. IEEE Trans. Med. Imag. 24(9), 2005.

10 15 20 25 300

1

2

3

4

5

Initial displacement (mm)

Rot

atio

n er

ror (

degr

ees)

+std

rms

-std

Clinically useful level

(b)

5

5 10 15 20 25 300

0.5

1

1.5

2 2.5

Initial displacement (mm)

Tran

slat

ion

erro

r (m

m)

+std

rms

-std

Clinically useful level

(a)

Medical Information Visualisation - BioMedical Visualisation (MediViz 2007)0-7695-2904-6/07 $25.00 © 2007