Non-Rigid Registration of 3D Surfaces by Deformable 2D Triangular Meshes

Arman SavranElectrical and Electronics Engineering DepartmentBogazici University, 34342 Bebek, Istanbul, Turkey

arman.savran@boun.edu.tr

Bulent Sankurbulent.sankur@boun.edu.tr

Abstract

Non-rigid surface registration, particularly registrationof human faces, finds a wide variety of applications incomputer vision and graphics. We present a new auto-matic surface registration method which utilizes both at-traction forces originating from geometrical and texturalsimilarities, and stresses due to non-linear elasticity of thesurfaces. Reference and target surfaces are first mappedonto their feature image planes, then these images are reg-istered by subjecting them to local deformations, and fi-nally 3D correspondences are established. Surfaces are as-sumed to be elastic sheets and are represented by triangularmeshes. The internal elastic forces act as a regularizer inthis ill-posed problem. Furthermore, the non-linear elas-ticity model allows us to handle large deformations, whichcan be essential, for instance, for facial expressions. Themethod has been tested successfully on 3D scanned humanfaces, with and without expressions. The algorithm runsquite efficiently using a multiresolution approach.

1. IntroductionNon-rigid registration of 3D surfaces is encountered in a

variety of applications. For human face processing it is anespecially crucial intermediate step. Among applicationsof non-rigid registration we can mention one-to-one corre-spondence between faces with the purpose of face recon-struction or recognition based on statistical face models [1].In computer animation, 3D characters can be animated us-ing available expression faces, or from recorded 3D facialvideos that are captured by recently developed 3D sensors[2, 7]. Other applications are transfer of textures, surfacedetails and animation controls between objects as well asgenerating morphing animations.

Non-rigid surface registration, however, is not trivial,like many other registration problems. Methods using thesupport of fiducial points have been used in many works.Praun et al. [4] search for a common parameterization oftwo objects using manually selected landmarks. In the work

of Allen et al. [3], a high resolution template mesh is opti-mally fitted to detailed human body range data with sparse3D markers. Yin et al. [5] also fit a template face modelto obtain dense point-to-point correspondence between ac-quired 3D faces. However, they do not use fiducial points,instead for each vertex of the template, they search for sim-ilar and close vertices of the target mesh. This algorithm isable to track facial expressions in a sequence of increasingvalence values.

A different approach, usage of generalized multidimen-sional scaling algorithm, is proposed for the dense corre-spondence estimation problem by Bronstein et al. [6]. Theyclaim that if objects are approximately isometric, one ofthem can be registered to another by finding correspondingsurface points that have similar geodesic distances. The ad-vantage of their method is that no initial pose alignment orfeature point extraction are required, and can run very effi-ciently. For example, 100 points over a 3D facial expressionsequence can be tracked successfully at a rate much fasterthan a second per frame.

In general, algorithms that work in 3D are computation-ally very intensive, their multiresolution implementation isdifficult, and require proper initial alignment. Some authorshave found that registration performed in 2D resolves someof the handicaps of 3D processing. Blanz et al. [1] em-ployed optical flow over the cylindrical projection of textureand range images of 3D faces for correspondence estima-tion. They incorporate also prior information coming fromprevious registrations of the database.

In [7] Wang et al. applied harmonic parameterizationto map 3D scanned faces onto image planes where featurecorners appearing on the texture and curvature images aretracked at a high video rate. An iterative refinement yieldsadequate correspondences for facial deformations. Litke etal. [8] also work in the parameterized domain which is op-timized with respect to length, angle and area distortions.Their sophisticated registration algorithm is based on vari-ational principles and aims to register faces with differentidentity or expression. This algorithm minimizes imagematching, regularization and feature demarcation energies

978-1-4244-2340-8/08/$25.00 ©2008 IEEE

to estimate deformations in the faces. Feature demarcationsare obtained by manually segmenting eyes, mouth and sym-metry lines to achieve accurate estimations.

In this paper, we propose the use of deformable 2D trian-gular meshes for non-rigid registration of 3D surfaces. Weassume that the two surfaces are elastic membranes and tryto fit one onto the other by deforming as little as possible.We prefer to carry out this task in 2D instead of 3D space forreasons mentioned above. Therefore, the two membranesare first deformed onto a plane, surface attributes are trans-ferred and than 2D one membrane is deformed to match theother one. Our method resembles the work in [8] where im-age processing techniques are employed to register mappedgeometry images by taking induced surface deformationsinto account. In contrast, we represent mapped surfaces bytriangular meshes and non-linear elasticity is directly pro-vided by the minimization of Green-Lagrange strain ten-sor over the triangle elements. The algorithm is compu-tationally less demanding and successful registrations areobtained without using manually determined features. Wedemonstrate our results on surfaces that bear relatively highdeformations that can be difficult to estimate, such as facesthat differ considerably in individual shapes or due to inten-sive expressions.

2. Mapping Surfaces onto Image PlanesThe simplest way to map piecewise planar surfaces onto

image planes could be projecting their vertices onto planes.For instance, Blanz and Vetter applied [1] cylindrical pro-jection to subsequently perform correspondence estimationin 2D. However, such projections may cause large distor-tions in the regions where the angle between the surfacenormal and the projection direction is large. Worse still,a bijective mapping is not guaranteed, which means thattriangles may overlap, thus leading to loss of informationduring mappings between 2D and 3D. These cause miss-ing correspondences, which should be handled in a post-processing step. To overcome these problems in this study,mesh parameterization is applied instead of any kind of pro-jection and smooth bijective mappings are obtained. Actu-ally, this process is nothing but the deformation of surfacesonto planes.

Many algorithms have been proposed to parameterize2D-manifold surfaces represented by 3D triangular meshes,mostly for texture mapping purposes. Briefly these al-gorithms are categorized according to mapping distortioncriterion that they minimize [9, 10]. Among the length-preserving, angle-preserving and area-preserving param-eterization approaches, least squares conformal mapping(LSCM) technique [11], which is an angle-preserving (con-formal) method, have been chosen in this study. Othermethods could also have been employed, however, con-formal methods are generally good at producing consistent

Figure 1. Mapped textures images (a, b) and mean curvature im-ages (c, d) of surfaces that are displayed in Figure 3.

parameterization, even with highly irregular triangulation[12], and unique solutions can be obtained. Furthermore,with LSCM, solution can be obtained very rapidly via lin-ear algorithms like conjugate gradient. Finally, LSCM hasthe advantage of being a natural boundary technique andrequiring only two constrained points.

Once we obtain mapping between 2D planes and sur-faces in 3D space, various attributes of the surface, like nor-mals, curvature or texture can be discretized in the imageplane. This re-sampling can be implemented by interpolat-ing between pixel attributes at the mesh vertices. For highdensity meshes, this is achieved very efficiently by utilizinggraphics hardware. For each image pixel, barycentric coor-dinates are determined and then new pixel values are inter-polated from the triangle vertices; or in the case of texture,bilinear interpolation is employed to sample from the tex-ture image at the barycentric coordinates. Sample mappedtexture and mean curvature images are shown in Figure 1.

3. Registration via Triangular Mesh Deforma-tion in 2D Domain

After obtaining adequate mappings of the two surfacesonto the image plane, the surface registration problem re-duces to 2D image registration problem and can be solvedvery efficiently. Lets denote the reference surface as Awhich is to be mapped to a target surface B. Their pla-nar parameterizations yield 2D domains DA, DB ⊂ R2.Hence, if we can find a mapping from DA to DB , this leadsto the mapping between the surfaces in 3D space. However,in general, it is not realistic to expect a bijective mappingbetween the surfaces, because the actual correspondencecan be partial and the mapping is not required to be onto.

Therefore, an injective mapping is obtained with this regis-tration scheme.

Registration of a reference to target is realized by match-ing the discretized surface attributes, which are mean cur-vature and gray-scale (average of color) texture images forthis study. However, it is well known that image registrationis an ill-posed problem. To regularize the ill-posed regis-tration problem smoothness information of the deformationfield can be incorporated in the solution. In this work, non-rigid deformation is estimated using deformable triangularmeshes that are thought as elastic sheets deforming accord-ing to stresses induced by image matching errors. We useGreen-Lagrange strain tensor for regularization, since it isappropriate for non-linear elastic behavior and large elasticdeformations can thus be handled. In the following subsec-tions image matching with triangular meshes, elastic defor-mations, and the multi-resolution registration algorithm areexplained.

3.1. Image Matching with Triangular Meshes

A mapping fromDA toDB can be expressed via a vectorfield

φ(p) = p + d(p), (1)

where p(u, v) ∈ DA denotes the 2D image coordinates andd(p) is a constant displacement. We apply finite elementdiscretization with triangular elements to approximate thisvector field. Thus, for each triangle t of the reference mesh,we have a mapping function q = φt(p) that maps a pointp in a triangle of the reference to the point q in the target.This function interpolates the mapped coordinates of the tri-angle vertices qk by

q = φt(p) =3∑

k=1

bk(p)qk (2)

where barycentric coordinates are obtained by

b1(p) = Area(pp2p3)/At,

b2(p) = Area(pp3p1)/At,

b3(p) = Area(pp1p2)/At,

(3)

and At is the area of triangle t.Thus, an image matching energy, EM , which accounts

for the total square matching error over domain DA be-comes

EM (φ) =12

∫p∈DA

‖IB(φ(p))− IA(p)‖2W dp

=12

∑t∈T

∫p∈Dt

‖IB(φt(p))− IA(p)‖2W dp(4)

where T is the set of triangles, Dt is the domain of trian-gle t, IB and IA represent multi-modal image values, and

W is a weighting matrix. Note that here, to exploit dif-ferent surface attributes during matching, weighted sums ofimage attributes are used by means of diagonal weighting.The weights are determined according to both variance andimportance of the channels.

The driving forces that deform a reference mesh to a tar-get come from the minimization of the image matching en-ergy. This energy can be minimized by using the gradientsat each mesh node n with respect to its mapped coordinatesqn = φ(pn). The gradient at vertex n is obtained throughthe chain rule as

∂EM

∂qn=∑t∈Tn

∫Dt

bk(t,n)(p)

(∂IB(q)∂q

∣∣∣∣φt(p)

)T

W et(p)dp

where et(p) = IB(φt(p))− IA(p).(5)

Here, Tn is the set of triangles connected to the node n,k(t, n) is the k. vertex of triangle t that corresponds to noden, and bk(t,n)(p) is thus the k. barycentric coordinate forthe point p.

The gradients are evaluated at each node of the meshto update the mapping vectors, φ(pn), in a gradient de-scent scheme. The image gradients are evaluated using3x3 Scharr masks [14]. We approximate the integrals inequations (4) and (5) by sampling at the recursively subdi-vided triangle centers. This sampling procedure, however,adapted to the area of triangles, since mesh triangles can dif-fer largely in area. In this way, while unnecessary computa-tions for small triangles are avoided, integrals over the largetriangles can be approximated accurately. This is achievedby subdividing triangles proportionally to their area. Bilin-ear interpolation is used for re-sampling from discrete im-ages.

3.2. Nonlinear Elastic Deformation

In finite element based registration methods, commonly,the regularization issue is handled by linear elastic model-ing, which in turn leads to simple linear solutions. However,many deformations cannot be accounted for by linear elas-tic models, which are, in general, convenient only for smalldeformations that can be approximated by linear models.However, we have to register surfaces subject to large de-formations, for instance faces with intensive expressions.Therefore, we propose to use a non-linear model, namelySt. Venant Kirchoff material model [13], to give requiredelastic behavior to triangular meshes. It is actually the sim-plest nonlinear hyperelastic material model.

The potential energy for St. Venant Kirchoff material isgiven by

W (E) =λ

2(trE)2 + µtrE2 (6)

where λ and µ are the Lame material constants, and E is theGreen-Lagrange strain tensor defined as

E =12

(∂φ

∂p

T∂φ

∂p− I

)(7)

=12(5dT +5d +5dT5 d

). (8)

In equation (8) the nonlinear relationship between displace-ment d and strain E is in the last term. Omitting this lastterm results in linear models. The advantage of nonlinearmodel is that, Green-Lagrange strain tensor is independentof rigid body motions, and thus measures pure deforma-tions. Therefore, it is adequate for large deformations.

We use minimization of the Green-Lagrange strain ten-sor to attain elasticity. The deformation energy is defined asthe Froebenius norm of this strain,

ED(φ) =∑t∈T

At‖E|2F

=14

∑t∈T

At

((a− 1)2 + 2b2 + (c− 1)2

)where a =

∥∥∥∥∂φt∂u

∥∥∥∥2

, b =∂φt∂u

T

.∂φt∂v

, c =∥∥∥∥∂φt∂v

∥∥∥∥2

(9)

where At is one of the mesh triangle area as in Equa-tion (3), and the other terms are quantities belonging to tri-angle t. This Froebenius norm corresponds to equation (6)with λ = 0 and µ = 1. Actually for a physical object µ > 0and λ+ µ > 0, and λ determines deformations in directionorthogonal to external forces, e.g. when we squeeze a rub-ber in horizontal direction it can widen in vertical direction.However, here the purpose is not correct physical modelingbut regularization.

Analytic expressions for gradients of ED with respect tovertex coordinate mappings must be obtained for the energyminimization task. With triangular mesh elements non-linearities are not too complex and can be solved directlywith a gradient descent scheme. During this minimization,stresses at each node of the mesh deform its neighboringtriangles to regularize the displacements due to forces gen-erated by image matching errors.

3.3. Multiresolution Registration Algorithm

The total energy minimized over the reference triangula-tion T is

ET (φ) = EM (φ) + ρED(φ). (10)

In this equation ρ controls the amount of elasticity and thusthe regularization during the registration. We minimize thetotal energy via gradient descent methods in a coarse-to-fineapproach. This strategy is very popular for registration since

Figure 2. Meshes generated over mapped domain of the referencesurface as in Figure 3. From (a) to (d) the number of triangles aredecreased according to image resolution.

local minimums can be avoided and faster convergence withless computational burden can be achieved.

To solve the problem in different scales, first, Gaussianimage pyramids are constructed one at the reference, theother at the target. Second, at each resolution, meshes aregenerated adapted to the scale of the pyramid level. How-ever, these meshes are produced over the reference domain,i.e., do not cover the whole image domain. This eliminatesthe need for image extrapolation outside the boundary aswell as reduces the computational load.

To produce meshes, first, constrained Delaunay triangu-lation is performed [15]. The boundary vertices, are usedas constraints in this triangulation. Feature points can alsobe used as additional constraints. Next, a mesh generationalgorithm which limits the maximum allowed triangle edgelength is applied [16] to the triangulation. In this study,this parameter is set according to the diagonal length, s, ofthe whole image domain rectangles. Thus, proper meshesfor each resolution can be obtained. Sample meshes with0.02 × s maximum edge length are shown in Figure 2 fordifferent pyramid resolutions. Notice that these meshes canbe quite irregular and the boundary triangles are relativelyvery small in coarser levels. However, since adaptive sam-pling is applied during calculations, this does not cause aserious loss of efficiency.

Registration starts from the coarsest level, and estimateddeformations are subsequently transferred to a finer level.The deformation transfer from one mesh to another is re-alized by barycentric mapping. For each node of a finerlevel, its barycentric coordinates at the coarser mesh arecalculated, and thus mapping of that node is obtained bya weighted sum of the previous mesh node values. Energy

Figure 3. Results of non-rigid registration of faces of two subjectsare shown by transferring the texture of the reference face to thedeformed faces. Both the deformed registration and halfway de-formation results are displayed.

minimization restarts in one higher resolution level initial-ized with the values of the previous step. This procedurecontinues until the finest scale and final deformation esti-mates are found.

4. Results

We tested our method presented above on faces acquiredwith structured light-based 3D digitizers. In Figure 3 reg-istration of two faces from the BU-3DFE database [17] isshown. Registration results of neutral and smiling facesfrom the Bosphorus database [18], is given in Figure 3.In these figures, textures of reference surfaces is mappedto that of deformed faces for comparison. One can ob-serve that facial features like eyes, eyebrows and lips havegood matching accuracy. Also, the overall deformations aresmooth. This can be best observed in the middle face whichdepicts the average of the reference and target faces. Furtherresults are given in Figure 5 based on faces from Bosphorusdatabase, where one reference face is registered to differentsubjects and different expressions. Again one can see thatthe features are matching well.

In these experiments, surface images were discretizedonto 512x512 resolution images, and four-level image pyra-

Figure 4. Results of non-rigid registration of a neutral face andsmiling face where the texture of the reference face is transferredto the deformed faces. Both the deformed registration and halfwaydeformation results are displayed.

mids were employed. For the image matching, usuallyequal importance was given to texture and shape channels;but better expression registration with texture and betteridentity registration with curvature channels could be ob-tained for some of the faces. It was observed that after con-vergence at coarser levels, deformations at the finest levelwere insignificant, making this finest resolution level redun-dant. Actually this is a rather expected result since major fa-cial features have low frequency characteristics. However,it was also observed that, in many cases, at the coarsest res-olution (64x64) level there were only few iterations. Thereason for this can be, probably, the good alignment of faceson the image planes.

Our algorithm is quite efficient. Many of the heavy cal-culations, like re-sampling for the reference meshes, aredone in the initialization step, and iterations are mostly per-formed on the coarser resolutions. Except for the mappingsbetween 3D and 2D, the computations do not depend on themesh resolution. In the experiments, the number of itera-tions till convergence were in between 50 and 200, and totalregistration duration (for mappings and energy minimiza-tion) changed from three to seven seconds using 3.0GHzCore2Duo processor.

Figure 5. Registration results of a reference face to faces of dif-ferent subjects and with different expressions. The faces at theleftmost and rightmost columns are the target faces. Texture ofreference is transferred to the deformed surfaces.

5. Conclusion and Future Work

A new automatic algorithm to solve the non-rigid regis-tration problem of two surfaces is presented. Registration isrealized by deforming one surface onto the other one aftermapping them onto a plane. This is achieved by first using abijective mapping, and then utilizing finite element methodto obtain the matching deformations. The differences ofgeometric and texture related surface attributes generatesdeformation forces, while elastic stresses regularize the re-sulting deformations. Nonlinear elasticity is used to copewith the large deformations by minimizing Green-Lagrangestrain tensor. Due to triangular mesh elements that are over-laid over a 2D domain, the calculations for energy gradi-ents become efficient. Also, following a multiresolution ap-proach, good results can be obtained in a short time.

We demonstrate our results on human face registration,which is still a challenging problem. Though in many facialexpressions correspondence estimations work well, how-ever, currently the algorithm is not designed to handle ex-pressions with open mouth, since the topology of the sur-face changes. As a future work we plan to study this is-sue, which presently can be solved with manual assistance.We also plan to experiment using different surface attributesand investigate better strategies to combine these modali-ties.

References[1] V. Blanz, and T: Vetter. A morphable model for the synthesis

of 3-D faces. Proc. SIGGRAPH’99, Los Angeles, USA, 187–194, 1999.

[2] L. Zhang, N. Snavely, B. Curless, and S.M. Seitz. Spacetimefaces: high resolution capture for modeling and animation.ACM Transactions on Graphics, 23(3):548–558, 2004.

[3] B. Allen, B. Curless, and Z. Popovic. The space of humanbody shapes. ACM Trans. on Graph., 22(3):587–594, 2003.

[4] E. Praun, W. Sweldens, and P. Schroder. Consistent meshparam. Proc. ACM SIGGRAPH, 179–184, 2001.

[5] L. Yin, X. Wei, P. Longo, and A. Bhuvanesh. Analyzing facialexpressions using intensity-variant 3D data for human com-puter interaction. 18th IAPR Int. Conf. on Pattern Recognition(ICPR 2006), Hong Kong, 1248–1251, 2006.

[6] A. Bronstein, M. Bronstein, and R. Kimmel. Calculus ofnonrigid surfaces for geometry and texture manipulation.IEEE Transactions on Visualization and Computer Graphics ,135:902–913, 2007.

[7] Y. Wang, M. Gupta, S. Zhang, S. Wang, X. Gu, D. Samaras,and P. Huang. High resolution tracking of non-rigid 3D mo-tion of densely sampled data using harmonic maps. in Proc.IEEE ICCV 2005, Beijing, China, 2005.

[8] N. Litke, M. Droske, M. Rumpf, and P. Schroder. An imageprocessing approach to surface matching. Eurographics Sym-posium of Geometry, 1–10, 2005.

[9] K. Horman, B. Levy and A. Sheffer. Mesh Param.:Theory andPractice. Siggraph Course Notes, Aug. 2007.

[10] M.S. FLoater and K. Hormann. Surface parameterization:a tutorial and survey. Advances in Multiresolution for Geo-metric Modelling, Mathematics and Visualization, Springer,157–187, 2005.

[11] B. Levy, S. Petitjean, N. Ray and J. Maillot. Least squaresconformal maps for automatic texture atlas generation. ACMTransactions on Graphics, 21(3):362–371, 2002. Proceedingsof SIGGRAPH 2002.

[12] M. Desbrun, M. Meyer, and P. Alliez. Intrinsic parameteri-zations of surface meshes. Proc. Eurographics, 2002.

[13] M. Bro-Nielsen. Medical image registration and surgery sim-ulation. IMM-DTU Ph.D. Dissertation, 1996 .

[14] J. Weickert and H. Scharr. A scheme for coherence-enhancing diffusion filtering with optimized rotation invari-ance. Jornal of Vis. Comm. Im. Rep., 13:103–118, 2002.

[15] J. D. Boissonnat, O. Devillers, S. Pion, M. Teillaud, andM. Yvinec. Triangulations in CGAL. Comput. Geom. The-ory Appl., 22:5–19, 2002.

[16] J. R. Shewchuk. Mesh generation for domains with smallangles. Proc. 16th Annu., AACM Sympos. Comput. Geom.,1-10, 2000.

[17] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rasato. A 3Dfacial expression database for facial behavior research. 7thInt. Conf. FGR06, 211–216, April 2006.

[18] A. Savran, N. Alyuz, H. Dibeklioglu, O. Celiktutan, B. Gok-berk, L. Akarun and B. Sankur. Bosphorus database for 3Dface analysis. First European Workshop on Biometrics andIdentity Management Workshop BioID, 2008.