DEFORMABLE REGISTRATION WITH SPATIALLY VARYING DEGREES OF FREEDOMCONSTRAINTS

James V. Miller, Girish Gopalakrishnan, Manasi Datar, Paulo R.S. Mendonca, Rakesh Mullick

Imaging Technologies, GE Global Research

ABSTRACT

Intrasubject deformable registration applications, such as lon-gitudinal analysis and multimodal imaging, use a high degreefreedom deformation to accurately align soft tissue. However,smoothness constraints applied to the deformation and insuf-ficient degrees of freedom in the deformation may distort themore rigid tissue types such as bone. In this paper, we presenta technique that aligns rigid structures using rigid constraintswhile aligning soft tissue with a high degree of freedom de-formation.

Index Terms— Deformable image registration, opticalflow, anatomy-specific constraints, volume preservation.

1. INTRODUCTION

Intrasubject image registration is a powerful tool for estimat-ing coordinate frame mappings in longitudinal sequences,time varying (4D) acquisitions, interventional imaging, andmultimodal imaging. The coordinate frame mappings al-low the clinician to fuse information coming from differ-ing sources, imaging anatomy, function or select pathology.When imaging the thorax or abdomen, deformable registra-tion is needed to accurately estimate the coordinate framemappings between soft tissue types in order to correct forshape change due to respiration. However, the abdomen andthorax contain a variety of tissue types with widely varyingrigidity. Individual structures, such as bone, predominantlymove rigidly with changes in patient position and posturewhile surrounding soft tissue deforms non-rigidly. Whiledeformable registration is capable of modeling rigid motion,most deformable methods use either too few degrees of free-dom in modeling the coordinate frame mapping, making itdifficult to accurately model proximal rigid and nonrigid mo-tion, or use too many degrees of freedom in modeling thecoordinate frame mapping, resulting in an overfitting of sen-sor variations on rigid structures and unrealistic local minimamappings.

A variety of approaches have been published to addressboth rigid and nonrigid motion leveraging image registration.Little et al. [1] present a thin-plate spline method to definenonrigid coordinate frame transformations where portions ofthe transform are constrained to be linear (affine). In another

method presented by Staringet al. [2] adaptively filter a de-formation field to apply rigidity constraints during regulariza-tion. Ruanet al. [3] regularize a dense deformation field suchthat the Jacobian in rigid regions is nearly orthogonal. DuBois d’Aische [4] explicitly models articulated structures andpropagates the deformations outside the articulated structuresusing a linear elastic model.

In a framework similar to some of the techniques referredabove, we propose modeling a coordinate frame mappingusing a dense deformation field and incorporating anatomi-cal information. We explicitly enforce linear and rigid bodyconstraints over large image areas as part of the deformationfield regularization. As the optimization process iterates, re-gions under linear and rigid body constraints always movelinearly or rigidly while unconstrained regions move undera free-form deformation. We demonstrate the approach us-ing a modified Demons style algorithm, although the basicapproach could be applied to many of the above techniques.

2. DEMONS

The image registration problem of two images is to estimatethe transformationT that maps a positionpf in a fixed imageIf to the corresponding positionpm = T (pf ) in the mov-ing imageIm. Thirion [5] introduced the Demons registra-tion algorithm to align images on the basis of optical flow.In the Demons algorithm, an iterative solution forT is givenby Ti+1 = ∂Ti Ti, where∂Ti is an incremental transfor-mation based on a set of positionspf1 , · · · , pfN

and a set ofincremental displacements~f(pf1), · · · , ~f(pfN

),

∂Ti = F(pf1 , · · · , pfN

, ~f(pf1), · · · , ~f(pfN))

,

(1)

~f(pf ) =Im(Ti(pf ))− If (pf )

‖∇If (pf )‖2 + (Im(Ti(pf ))− If (pf ))2∇If (pf ).

(2)

The functionF () in equation 1 allows a diverse set of in-cremental transformations∂Ti to be modeled, ranging fromfree-form deformations to parameterized transformations. Inthe former, a dense set of incremental displacement vectors isestimated. This is how the Demons algorithm is implemented

in the Insight Segmentation and Registration Toolkit [6]. Inthe latter, parameterized transforms, such as affine or B-splinetransformations, are estimated from the positions and incre-mental displacement vectors. This latter approach makesthe Demons algorithm similar to parameter estimation ap-proaches to image registration. The functionF () can also beused to regularize the displacement field.

In this paper, we explore using transformationsTi and in-cremental transformations∂Ti whose representation and de-grees of freedom vary spatially. Here, some positions inIf

will be governed by rigid body parameterized transformationswhile other positions inIf will be governed by free-form de-formations.

3. SPATIAL CONSTRAINTS ON DEFORMATIONDEGREES OF FREEDOM

Our approach is to delineate regions,Ωj , in the fixed imageIf

and assign a transformation modelΘj to each region. Regiondelineations can be derived from an atlas, can be the resultof automated and semi-automated segmentation algorithmsapplied to individual structures, or defined manually. Whenregistering CT imagery without injected contrast, large rigidbone structures can be easily identified using simple thresh-olding techniques followed by connected component tech-niques to isolate individual structures.

For each region, we prescribe a transformation model togovern its motion. Currently, we provide rigid, linear (affine),and free-form transformations. Other parameterized andspline based transformations are possible. The transformationmodel for each region is used to regularize the deformationfield in a Demons style algorithm. The registration algorithmis as follows:

1. Initialize deformation field,T0 ← ~0

2. Estimate incremental displacements,

~f(pf ) =Im(Ti(pf ))− If (pf )

‖∇If (pf )‖2 + (Im(Ti(pf ))− If (pf ))2∇If (pf )

(3)

3. Composite the incremental displacements into the cur-rent deformation field,Ti+1 = Ti + ∂Ti

4. Regularize each regionTi+1,Ωjbased on its transfor-

mation modelΘj

(a) Θj ∈ free-form deformation:

Ti+1,Ωj← Ti+1,Ωj

∗N(0,Σ) (4)

(b) Θj ∈ linear,Θj ∈ rigid:

Θ∗j = argminΘj

‖F (Ti+1,Ωj; Θj)− Ti+1,Ωj

‖2

(5)

Ti+1,Ωj← F

(Ti+1,Ωj

; Θ∗j

), (6)

whereTi+1,Ωj is the portion of the deformation fieldunder the regionΩj .

5. Repeat with step 2

In the regularization step 4, we use the standard gaussiansmoothing approach to yield a smoothly varying deformationfield for the regionsΩj whose transformation modelsΘj arefree-form transformations. For other transformation types, weestimate the parameters of the transformationΘj that best de-scribes (under an appropriate metric) regionΩj ’s deformationfield Ti,Ωj . Then, we projectTi,Ωj onto the space spanned bythe transformationΘj . We then continue with Demons iter-ations until convergence. Note, that at each iteration, the de-formation field is explicitly constrained to the feasible regionprescribed by the region delineationsΩj and the prescribedtransformation typesΘj . Thus, regions that are defined tomove rigidly will move rigidly during the entire optimizationprocess, yielding valid deformations at each iteration. This isin contrast to other techniques, where the rigidity constraintsmay only be fully satisfied at the optimum. The transforma-tion estimation and projection operations are described in thefollowing sections.

4. ESTIMATING LINEAR TRANSFORMATIONS

A positionpm in Im is related to a positionpf in If througha transformation as well as through a displacement vector:pm = T (pf ), pm = pf + ~f(pf ). Combining these equa-tions, each displacement vector can be written as a functionof the transformation~f(pf ) = T (pf ) − pf . If T is a lineartransformation,~f(pf ) can be written in matrix form~f(pf ) =

B

[pf

1

]. Given a set of measurement locationspfi and a

set of displacement vectors~f(pfi), we construct the linear

multivariate model

Y = XBT + U (7)

whereX andY are matrices whose rows are composed of thepositionspf1 , · · · , pfN

and the displacement vectors~f(pf1),· · · , ~f(pfN

)

X =

pTf1

1...

...pT

fN1

Y =

~f(pf1)

T

...~f(pfN

)T

(8)

with the matrixU modeling the noise. When the noise ismultivariate normal, the maximum likelihood estimator forBis given by [7]

BT =(XT X

)−1XT Y. (9)

To constrain the original set of displacement vectors to thespace spanned by the estimated linear transformation, the vec-

tors ~f(pfi) are replaced with

~f(pfi) = B

[pfi

1

]. (10)

We use equation 9 as the implementation of equation 5, es-timating the parametersΘj of the linear transformation for aregionΩj prescribed to move linearly. We use equation 10as the implementation of equation 6, projecting the displace-ment vectorsTi,Ωj onto this space, thereby enforcing the lin-ear constraints.

Equation 9 provides the maximum likelihood estimateof the linear model, assuming all measurements are trustedequally. If the displacement vectors have varying certainty, aweighted estimator can also be used, where equations 7 and9 become

WY = WXBT + WU, (11)

BT =(XT WT WX

)−1XT WT WY. (12)

Equation 12 provides a mechanism to employ robust lossfunctions and M-estimators [8] to estimate the parameters ofthe linear model. Using M-estimators will increase the ro-bustness of the linear model estimated from the displacementvectors.

5. ESTIMATING RIGID BODYTRANSFORMATIONS

The linear modelB estimated in equation 9 maps positionsin If to displacement vectors that model an underlying affinemotion of the positions. To model rigid body motion, addi-tional constraints are needed on the parameters of the transfor-mation. We modify the problem formulation from the linearcase such that the goal is to estimate the rigid body transform(R, t) that maps positions inIf to positions inIm under rigidbody motion. Here, equation 5 becomes

R∗, t∗ = argminR,t‖Y −XR− t‖2 (13)

subject to : RT R = I (14)

where the matrixX in this case has dropped the final columnof 1’s and the matrixY contains positionspfi

+ ~f(pfi) in

Im instead of the displacement vectors. A region of the de-formation fieldΩj constrained by rigid motion is regularizedby solving equation 13 under the constraints 14 and project-

ing the vectors~f(pfi) in Ωj with ~f(pfi

) = Rpfi+ t − pfi

.The constrained optimization problem in equations 13 and 14operates in the Special Euclidean Group, SE(3). Horn [9]presents a closed form solution which changes the parame-terization of the problem to use quaternions. The change ofparameterization converts the optimization problem in equa-tions 13 and 14 to a well known eigenvector problem. Severalextensions to Horn’s basic method have been proposed as the

solution step iterated in Iterative Closest Point (ICP) regis-tration [10, 11, 12]. Many of these extensions can also be ap-plied here, ignoring the iteration of ICP. In particular, Fitzgib-bon [10] presents a formulation whereby robust loss functionscan be incorporated into the metric, thereby increasing therobustness of the estimated rigid model to the displacementvectors.

6. RESULTS

We have tested our algorithm on both2D and3D real im-ages. We compare the Demons registration algorithm fromthe Insight Segmentation and Registration Toolkit (ITK) witha version of this algorithm modified to enforce linear and rigidbody constraints.

Figure 1 illustrates this technique applied to a2D CT im-age of the cervical region of the spine. The ground truth coor-dinate transformation between the image sets is a translationof 3mm×3mm×3mm. The registration without constraintsfails to align the complete rib or vertebrae whereas the regis-trations incorporating linear and rigid body constraints alignthe data well (Figure 1).

For the3D case, we performed registrations on respira-tory gated CT images that were obtained from4 different pa-tients. Each patient data comprised of6 respiratory-gated vol-umes that were then registered to a chosen reference gate. Intotal, we tested each algorithm on20 registrations(5 pairs×4patients). For the3D volumes, apart from visual comparisons(Figure 2), we calculate the Dice’s coefficient that measuresthe similarity between the bone-masks obtained from the al-gorithms with that obtained from the reference image. Forbinary masksA andB, DC(A,B) = 2|A∩B|

|A|+|B| . We observe aconsistent improvement in the Dice’s coefficient by using ourmethod. Table 1 shows the results for a patient [5 registra-tions].

DC(Demons) DC(Constraints)Pair-1 0.808146 0.853202Pair-2 0.808395 0.854671Pair-3 0.828938 0.874208Pair-4 0.82035 0.869335Pair-5 0.808686 0.859021

Table 1. Dice’s coefficient calculated between the bone-masks for patient #1 showing the improvement in overlap byusing rigid constraints on Demons

7. CONCLUSIONS

Deformable registration techniques are capable of modelinga wide range of nonrigid motion. However, when regions ofan image are known to move rigidly, these techniques maynot estimate true rigid motion. While deformable techniquescan model rigid body vector fields, intermediate results be-fore convergence and local minima results may not reflect re-

(a) (b)

Fig. 1. (a) Comparison of the contour of the constrained re-gion. (b) Comparison of mean square metric convergence.Red: Demons, Blue: Linear constraints, Green: Rigid con-straints

(a) Original CT Image (b) Rigid registration

(c) Demons registration (d) Constrained-Demons

Fig. 2. Overlays of the bone-mask surfaces to show thatthe constrained-demons fills the gap between deformable andrigid registrations.

alizable configurations. We have shown how a deformableregistration technique known for estimating high degree offreedom deformations (Demons) can be constrained to incor-porate rigid and linear motions embedded in free-form defor-mations. We achieve this by applying custom regularizationsto each image region. The regularizations for linearly andrigidly constrained regions estimate a low parameter model ofthe motion from the current Demons vectors and use the esti-mate to project the deformation field in that region to satisfythe constraints. This allows constraints to be applied to largeimage regions and ensures that the constraints are satisfiedthroughout the Demons iterations. The low parameter modelscan be standard maximum likelihood estimates or weightedestimates using robust loss functions. This approach has alsoshown a benefit in the rate of convergence (Figure 1(b)).

8. ACKNOWLEDGEMENTS

We wish to thank Valentino Bettinardi, Maria Carla Gilardi,Ferruccio Fazio from Hospitale San Raffaele, Milan for pro-viding the clinical data and Kris Thielemans from Hammer-smith Imanet-GEHC for his help.

9. REFERENCES

[1] J. A. Little, D. L. G. Hill, D. J. Hawkes, and L. Bridge,“Deformations Incorporating Rigid Structures,” inPro-ceedings of MMBIA, 1996, vol. 96, p. 104.

[2] M. Staring, S. Klein, and J. P. W. Pluim, “Nonrigid reg-istration using a rigidity constraint,” inProceedings ofSPIE, 2006, vol. 614413, pp. 1–10.

[3] D. Ruan, J. A. Fessler, M. Roberson, J. Balter, andM. Kessler, “Nonrigid registration using regularizationthat accommodates local tissue rigidity,” inProceedingsof SPIE, 2006, vol. 614412, pp. 1–9.

[4] A. Du Bois dAische, M. De Craene, B. Macq, andS. K. Warfield, “An improved articulated registrationmethod for neck images,” inProc. of Engineering inMedicine and Biology Conference, 2005.

[5] J. -P. Thirion, “Image matching as a diffusion process:an analogy with Maxwell’s demons,”Medical ImageAnalysis, vol. 2, no. 3, pp. 243–260, 1998.

[6] L. Ibanez, W. Schroeder, L. Ng, andJ. Cates, The ITK Software Guide,http://www.itk.org/ItkSoftwareGuide.pdf, secondedition, 2005.

[7] K. V. Mardia, J. T. Kent, and J. M. Bibby,MultivariateAnalysis, Probability and Mathematical Statistics. Aca-demic Press, New York, 1979.

[8] D. A. Forsyth and J. Ponce,Computer Vision: A ModernApproach, Prentice Hall Professional Technical Refer-ence, 2002.

[9] B. K. P. Horn, “Closed-form solution of absolute orien-tation using unit quaternions,”Journal of the OpticalSociety of America A, vol. 4, no. 4, pp. 629–642, 1987.

[10] A. W. Fitzgibbon, “Robust registration of 2D and 3Dpoint sets,” inBritish Machine Vision Conference, 2001,vol. 2, pp. 411–420.

[11] S. Granger and X. Pennec, “Multi-scale EM-ICP: Afast and robust approach for surface registration,” inECCV. 2004, Springer.

[12] A. Estepar, R. S. J. and Brun and C. F. Westin, “RobustGeneralized Total Least Squares Iterative Closest PointRegistration,” inMICCAI. 2004, pp. 234–241, Springer.