[Mathematics and Visualization] Innovations for Shape Analysis || Group-Valued Regularization for Motion Segmentation of Articulated Shapes

Chapter 12Group-Valued Regularization for MotionSegmentation of Articulated Shapes

Guy Rosman, Michael M. Bronstein,Alexander M. Bronstein, Alon Wolf, and Ron Kimmel

Abstract Motion-based segmentation is an important tool for the analysis ofarticulated shapes. As such, it plays an important role in mechanical engineering,computer graphics, and computer vision. In this chapter, we study motion-basedsegmentation of 3D articulated shapes. We formulate motion-based surface seg-mentation as a piecewise-smooth regularization problem for the transformationsbetween several poses. Using Lie-group representation for the transformation ateach surface point, we obtain a simple regularized fitting problem. An Ambrosio-Tortorelli scheme of a generalized Mumford-Shah model gives us the segmentationfunctional without assuming prior knowledge on the number of parts or even thearticulated nature of the object. Experiments on several standard datasets comparethe results of the proposed method to state-of-the-art algorithms.

G. Rosman (�) � R. KimmelDepartment of Computer Science, Israel Institute of Technology, 32000, Haifa, Israele-mail: [email protected]; [email protected]

M.M. BronsteinInstitute of Computational Science, Faculty of Informatics, Universita della Svizzera italiana,Lugano, Switzerlande-mail: [email protected]

A.M. BronsteinSchool of Electrical Engineering, Faculty of Engineering, Tel Aviv University, Ramat Aviv69978, Israele-mail: [email protected]

A. WolfDepartment of Mechanical Engineering, Technion – Israel Institute of Technology,Haifa, 32000, Israele-mail: [email protected]

M. Breuß et al. (eds.), Innovations for Shape Analysis, Mathematics and Visualization,DOI 10.1007/978-3-642-34141-0 12, © Springer-Verlag Berlin Heidelberg 2013

263

264 G. Rosman et al.

12.1 Introduction

Articulated objects segmentation is a key problem in biomechanics [1], mechanicalengineering, computer vision [8,30,38,41,54], and computer graphics [6,35,37,39,59, 64, 65]. Related problems of deformation analysis [4, 63] and motion segmen-tation [5, 22] have also been studied extensively in these disciplines. Algorithmssolving these problems try to infer the articulated motion of an object, given severalinstances of the object in different poses. Simultaneously, the segmentation ofthe object into rigid parts takes place along with motion estimation between thecorresponding parts in the various poses.

Most motion analysis techniques make some assumptions on the object to besegmented. These usually concern the number or location of rigid parts in thearticulated object. This can be in the form of a skeleton describing the topologyof the shape, or some other prior on the object structure. Such priors are usuallyformulated in an ad hoc manner, but not based on the kinematic model commonlyassumed for near-rigid objects [1,4]. In cases where such a prior is not available forthe objects in question, or where assumptions about the data are only approximate,this can lead to errors in the segmentation and motion estimation.

Another common assumption, especially in graphics applications, is that ofknown correspondences. In computer graphics, the problem is usually referred toas dynamic mesh segmentation. While a matching technique between poses can becombined with existing motion segmentation tools, a more complete formulationfor motion segmentation should handle the correspondence problem implicitly.

Clearly, the above assumptions are often too limiting in real-world applications,and should be avoided as part of the basic problem formulation. We would likeinstead to apply the intuition often used when studying real-life near-rigid objects,about the existence of a representative rigid motion existing for each body part. Wewish, however, to avoid detecting the articulated parts in advance. Furthermore, insome object, a clear partition into rigid parts may not exist for all of the surface.We wish to obtain reasonable results in such a case. In other words, we wouldlike to obtain a “soft” segmentation of the surface, without knowing the number orlocation of regions in advance, an explicit analysis of the surface features, or havingadditional priors on the various object parts. Also, we strive towards a formulationof motion segmentation that incorporates an implicit handling of the correspondenceproblem, given a reasonable initialization.

12.1.1 Main Contribution

In this chapter we try to remedy the shortcoming of existing approaches toarticulated motion estimation by combining the two tasks of motion estimationand segmentation into a single functional. This scheme has been described in arecent conference paper [50] and we now slightly expand upon it. Unlike existingmethods, we propose a principled variational approach, attempting to find a rigid

12 Group-Valued Regularization for Motion Segmentation of Articulated Shapes 265

transformation at each surface point, between the instance surfaces, such that theoverall transformation is described by a relatively sparse set of such transformations,each matching a rigid part of the object. The functional we propose regularizes themotion between the surfaces, and is guided by the fact that the parameters of themotion transformations

(i) Should describe the motion at each point with sufficient accuracy.(ii) Should vary smoothly within the (unknown) rigid parts.

(iii) Can vary abruptly between rigid parts.

We see our main contribution in these:

A new framework: First, we propose an axiomatic variational framework forarticulated motion segmentation. While focusing on the segmentation problem inthis chapter, our framework is more general and the proposed functionals can beeasily incorporated into other applications such as motion estimation, tracking,and surface denoising.Variational segmentation: We claim that using the right parameterization, takenfrom the specific domain of rigid motion analysis, we can formulate thearticulated motion segmentation problem as a generalization of classical toolsin variational computer vision. This allows for an elegant and simple solutionwithin the proposed framework, obtaining results competitive with domain-specific state-of-the-art tools.A novel visualization algorithm: Third, we suggest a spatially-coherent algorithmfor spatial visualization of group valued data on manifolds, which draws from thesame variational principles.

12.1.2 Relation to Prior Work

Several previous works have attempted motion based segmentation of surfaces.We mention but a few of these. Kompatsiaris et al. [38] use an estimation of therigid motion at each segment in order to segment the visible surface in a coarse-to-fine manner. Arcila et al. [6] iteratively refine the segmentation for segmentswhose transformation error is too large. Wuhrer and Brunton [64] use a dual treerepresentation of the surface with weights between triangles set according to thedihedral angles. Lee et al. [39] use a similar graph-based formulation, looking atdeformation matrices around each triangle.

The scheme we propose involves diffusing the transformations between posesalong the surface, in the spirit of the Ambrosio-Tortorelli scheme [2] for Mumford-Shah segmentation [43]. The diffusion component of our scheme is a diffusionprocess of Lie-group elements, which has recently attracted significant attentionin other applications [25, 28, 56]. In diffusing transformations on the surface, ourwork is similar to that of Litke et al. [40], although the parameterization of themotion and of the surface is different. In addition, we do not make an assumption


on the surface topology; to that end, the proposed method diffuses transformationsalong the surface, rather than representing the surface in an evenly sampled 2Dparametrization plane. When dealing with real-life deformable objects that seldomadmit regular global parametrization, such an assumption could be too restrictive.

The idea of combining soft segmentation and motion estimation has beenattempted before in the case of optical flow computation (see, e.g., [3,18]). In opticalflow fields, however, the motion field is merely expected to be piecewise smooth.For truly articulated objects one would expect piecewise-constant flow fields, whenexpressed in the correct parametrization.

Finally, the functional can be extended with priors from general mesh segmen-tation techniques. These are usually based on the geometry of the surface itself,and obtain remarkable results for a variety of objects. We point the reader to[9,21,36,55], and references therein, for additional examples of mesh segmentationalgorithms. We do not, however, use an additional prior as such an addition willprevent the isolated examination of the principles shown in this chapter.

12.2 Problem Formulation

We now proceed to define the problem we try to solve and the proposed model.

12.2.1 Articulation Model

We denote by X a two-manifold representing a three-dimensional shape. We assumeX to have several embeddings into R

3. Each of these embedding constitutes a poseof the surface. In the following, we will denote by x W X ! R

3 the embedding ofX into R

3, and use synonymously the notation x and x referring to a point on themanifold and its Euclidean embedding coordinates, for a specific pose.

In the setting of rigid motion segmentation, we assume that X represents anarticulated shape, i.e., it can be decomposed into rigid parts S1; : : : ; Sp. These aretransformed between different poses of the objects by a rigid transformation. Thistransformation, a rotation and a translation, is an isometry of R3. The rigid partsare connected by nonrigid joints J1; : : : ; Jq , such that X D Sp

iD1 Si [ Sq

kD1 Jk .An articulation Y D AX is obtained by applying rigid motions Ti 2 Iso.R3/

to the rigid parts, and non-rigid deformations Qk to the joints, such that AX DSpiD1 Ti Si [Sq

kD1 QkJk .

12.2.2 Motion Segmentation

The problem of motion-based segmentation can be described as follows: given twoarticulations of the shape, X and Y , extract its rigid parts. An extension to the case


of multiple shape poses is straightforward. We therefore consider in the followingonly a pair of shapes for the sake of simplicity and without loss of generality.A strongly related question attempts to determine, given these articulations, themotion parameters linking the poses of the object.

Assuming that the correspondence between the two poses X and Y is known,given a point x 2 X and its correspondent point y.x/ 2 Y , we can find a motion g 2G such that gx D y, where G is some representation of coordinate transformationsin R

3. This motion g may change, in the setting described above, for each surfacepoint. We therefore consider g to be a function g W X ! G. We will simultaneouslyuse gx 2 R

3 to denote the action of g.x/ on the coordinates of the point x, as wellas consider the mapping given by g W X ! G and its properties.

We note that typical representations of motion in R3 contain more than three

degrees of freedom. In this sense, they are over-parameterized [45], and thus somemeasure of regularity is required in order to avoid ambiguity as well as favor ameaningful solution. On the other hand, we note that since the articulated partsof the shape move rigidly, if we choose an appropriate motion representation (asdetailed below), two points x; x0 2 Si will undergo the same transformation, fromwhich it follows that g.x/jx2Si

D const. One possibility is to adopt a constrainedminimization approach, forcing g.X/ D Y , where g.X/ is a notation for the setg.x/x.x/ for all x 2 X . This approach, however, needs to somehow handle the setof joints, for which such a constraint may be meaningless. In general, restricting thefeasible set of solutions by such constraints or even constraints involving an error inthe data may be harmful for the overall result. In order to avoid this, another possibleapproach is to take an unconstrained, yet regularized, variational formulation,

mingWX!G

�ED.g/ C �.g/; (12.1)

where � denotes a smoothness term operating on the motion parameters field. Thisterm is expected to be small for fields g which are piecewise constant on themanifold X . While an appropriate parameterization of motion g, and regularizationterm �.g/ are crucial, we also require a data term that will encourage consistencyof the transformation field g with the known surface poses. Specifically, we wish tofavor a transformation field where the point x is taken by its transformation g.x/ to apoint on the other surface. ED.g/ is our fitting term which measures this consistencywith the data.

ED.g/ DZ

X

kg.x/x � y.x/k2da; (12.2)

where y.x/ 2 R3 denotes the coordinate of the point y.x/ 2 Y corresponding to

x, g.x/ is the transformation at x, and da is a measure on X . We have assumed inthe discussion so far that the correspondence between X and Y is known, which isusually not true. We can solve for the correspondence as part of the optimizationin an efficient manner. We will mention this issue in Sect. 12.4.1. We use the


term corresponding point y.x/ since, as in the case of iterative closest point (ICP)algorithms [13,67], several approaches for pruning erroneous or ineffective matchesexist [53].

Minimizing the functional with respect to g; y.x/ from a reasonable initialsolution allows recovery of the articulated parts by clustering g into regions of equalvalue. Yet another choice of a data term is a semi-local fitting term, is a semi-localone,

ED;SL.g/ DZ

X

Z

y2N .x/

kg.x/x0 � y.x0/k2da0da; (12.3)

where N .x/ denotes a small neighborhood around the point x (we took N .x/ to bethe 12 nearest neighbors). This fitting term, by itself, formulates a local ICP process.The functional (12.1) equipped with the semi-local data term can be consideredas the geometrical fitting equivalent of the combined global-local approach for opticflow estimation [17].

The simplest representation of motion is a linear motion model, affectivelysetting G to be the group of translation, or G D R

3. This results in the motionmodel gx D x C t D y for some t 2 R

3. However, such a simplistic model fails tocapture the piecewise constancy of the motion field in most cases. Instead of turningto a higher order approximation model such as the affine over-parameterized model[46], or to more elaborate smoothness priors [60], we look for a relatively simplemodel that will capture natural motions with a simple smoothness prior. Thus weturn to a slightly different motion model, naturally occurring in motion research.

12.2.3 Lie-Groups

One parametrization often used in computer vision and robotics [28, 37, 42, 61] isthe representation of rigid motions by the Lie-group SE.3/ and the correspondingLie-algebra se.3/, respectively. In general, given two coordinate frame, an elementin SE.3/ describes the transformation between them. Works by Brockett [14], Parket al. [47] and Zefran et al. [69, 70] strongly relate Lie-groups, both in their globaland differential description, to robotics and articulated motions. We give a very briefintroduction to the subject and refer the reader to standard literature on the subject(e.g., [29, 44]) for more information.

Lie-groups are topological groups with a smooth manifold structure such that thegroup action G � G 7! G and the group inverse are differentiable maps.

For every Lie-group, we can canonically associate a Lie-algebra g. A Lie-algebrais as a vector space endowed with a Lie brackets operator Œ�; �� W G�G!G,describing the local structure of the group. The Lie-algebra associated with aLie-group can be mapped diffeomorphically via the exponential map onto aneighborhood of the identity operator and its tangent space.


This property will allow us express neighboring elements in the Lie-group in avector space, and thereby define derivatives, regularity, and diffusion operators onthe group valued data.

In this chapter, we are specifically interested in the special orthogonal (rotation)matrix group SO.3/ and the Euclidean group SE.3/ to represent rigid motions. Thesecan be represented in matrix forms, where SO.3/ is given as

SO.3/ D ˚R 2 R3�3; RT R D I

�; (12.4)

and SE.3/ is given by

SE.3/ D��

R0 1

�

; R 2 SO.3/; t 2 R3

�

: (12.5)

The Lie-algebra of SO.3/, so.3/ consists of skew-symmetric matrices,

so.3/ D ˚A 2 R3�3; AT D �A

�; (12.6)

whereas the Lie-algebra of SE.3/ can be identified with the group of 4 � 4 matricesof the form

se.3/ D��

A0 0

�

; A 2 so.3/; t 2 R3

�

; (12.7)

where so.3/ is the set of 3 � 3 skew-symmetric matrices.In order to obtain piecewise constant description over the surface for the

relatively simple case of articulated object, we would like the points at each objectpart to have the same representative. Under the assumption of G D SE.3/, thisdesired property holds. We note, however, that the standard parameterization ofsmall rigid motions has six degrees of freedom, while the number of degreesof freedom required to describe the motion of point is mere three. Thus, thisparameterization clearly constitutes an over-parameterized motion field [46] forarticulated surfaces.

We now turn to the regularization term, �.g/, and note that the formulationgiven in Eq. (12.1) bears much resemblance to total variation (TV) regularizationcommon in signal and image processing [51]. Total variation regularization doesnot, however, favor distinct discontinuity sets. This property of TV regularizationis related to the staircasing effect. Furthermore, in the scalar case, discontinuitysets form closed curves, which may not be the case in some surfaces with largejoint areas. Instead, a model that better suits our segmentation problem is theMumford-Shah segmentation model [43]. This model can be implemented usingan Ambrosio-Tortorelli scheme [2], which can be easily generalized for the case ofmaps between general manifolds such as maps from surfaces into motion manifolds.We further describe the regularization chosen in Sect. 12.3.


We also note that due to the non-Euclidean structure of the group, special careshould be taken when parameterizing such a representation [28, 37, 42, 56], asdiscussed in Sect. 12.4.2.

12.3 Regularization of Group-Valued Functions on Surfaces

Ideally, we would like the transformation field defined on the articulated surface tobe piecewise smooth, if not piecewise constant. Therefore, a suitable regularizationof the transformation parameters is required. Since the Lie-groupG as a Riemannianmanifold, it is only natural to turn to regularization functionals defined on mapsbetween manifolds of the form g W X ! G.

A classical functional defined over such maps is the well-known Dirichlet energy[26],

�DIR.g/ D 1

2

Z

X

hrg; rgig.x/da D 1

2

Z

X

tr�g�1rg

�2da; (12.8)

where rg denotes the intrinsic gradient of g on X , h�; �ig.x/ is the Riemannianmetric on G at a point g.x/, and da is the area element of X . This functional is themore general form of the Tikhonov regularization (for Euclidean spaces X and G),and its properties are well defined for general manifolds, as studied by Eells [26].

Minimizers of the Dirichlet energy are called harmonic maps. These result froma diffusion process, and are often used for surface matching [62, 68].

12.3.1 Ambrosio-Tortorelli Scheme

Unfortunately, the Dirichlet energy favors smooth maps defined on X , whereasour desired solution has discontinuities at the boundaries of rigid parts. We would,intuitively, want to prevent diffusion across these discontinuity curves. This can beobtained by adding a diffusivity function v W X ! Œ0; 1� to the Dirichlet functional,leading to the generalized Ambrosio-Tortorelli scheme [2] for Mumford-Shahregularization [43].

�AT.g/ DZ

X

1

2v2hrg; rgig C �hrv; rvi C .1 � v/2

4�

!

da; (12.9)

where � is a small positive constant. This allows us to extend our outlook inseveral ways. The Mumford-Shah functional replaces the notion of a set of regionswith closed simple boundary curves with general discontinuity sets. It furthermoregeneralizes our notion of constant value regions with that of favored smoothness


inside the areas defined by these discontinuity curves. This is in order to handleobjects which deviate from articulated motion, for example in flexible regions orjoints.

Furthermore, the generalized Ambrosio-Tortorelli scheme allows us to explicitlyreason about places in the flow where the nonlinear nature of the data manifoldmanifests itself. Suppose we have a solution .g�; v�/ satisfying our piecewise-constancy assumptions of g, and a diffusivity function with 0 at region boundariesand 1 elsewhere. At such a solution, we expect two neighboring points whichbelong to different regions to have a very small diffusivity value v connecting them,effectively nullifying the interaction between far-away group elements which isdependent on the mapping used for the logarithm map at each point, and hencecan be inaccurate [32, 42]. While such a solution .g�; v�/ may not be a minimizerof the functional, it serves well to explain the intuition motivating the choice of thefunctional.

12.3.2 Diffusion of Lie-Group Elements

In order to efficiently compute the Euler-Lagrange equation corresponding to thegeneralized Ambrosio-Tortorelli functional (12.9), we transform the neighborhoodof each point into the corresponding Lie-algebra elements before applying thediffusion operator. Using Lie-algebra representation of differential operators forrigid motion has been used before in computer vision [56], numerical PDEcomputations [32], path planning and optimal control theory [37, 42].

The Euler-Lagrange equation for the generalized Dirichlet energy measuring themap between two manifolds is given as [26]

�X g˛ C � ˛ˇ�

˝rgˇ; rg�˛g.x/

D 0; (12.10)

where ˛; ˇ � enumerate the local coordinates of our group manifold, se.3/, and weuse Einstein’s notation according to which corresponding indices are summed over.� ˛

ˇ� are the Christoffel symbols of SE.3/, which express the Riemannian metric’slocal derivatives. We refer the reader to [24] for an introduction to Riemanniangeometry. Finally, �X denotes the Laplace-Beltrami operator on the surface X .

In order to avoid computation of the Christoffel symbols, we transform thepoint and its neighbors using the logarithm map at that point in SE.3/. Thediffusion operation is now affected only by the structure of the surface X . Afterapplying the diffusion operator, we use the exponential map in order to return tothe usual representation of the transformation. While this approach may suffer atdiscontinuities, where the logarithm and exponential maps are less accurate, it isat these continuities that we expect the diffusivity function v to be very small,preventing numerical instability. In practice, as we will demonstrate, this did notpose a significant problem.


Algorithm 1 Articulated surface segmentation and matching1: Given an initial correspondence.2: for k D 1; 2; : : : ; until convergence do3: Update gkC1=2; vkC1 w.r.t. the diffusion term, according to Eq. (12.12).4: Obtain gkC1 according to the data term, using Eq. (12.15).5: Update ykC1.x/, the current estimated correspondence of the deformed surface.6: end for

12.4 Numerical Considerations

We now describe the algorithm for articulated motion estimation based on theminimization of the functional

E.g; v/ D �EDATA.g/ C �AT .g; v/; (12.11)

where EDATA.g/ is the matching term defined by Eq. (12.2), and �AT .g; v/ isdefined in Eq. (12.9). The main steps of the algorithm are outlined as Algorithm 1.Throughout the algorithm we parameterize g.x/ based on the first surface, given asa triangulated mesh, with vertices fxi gN

iD1, and an element from SE.3/ defined ateach vertex. The triangulation is used merely to obtain a more consistent numericaldiffusion operator, and can be avoided, for example by point-cloud based Laplacianapproximations [10]. Special care is made in the choice of coordinates during theoptimization as explained in Sect. 12.4.2.

12.4.1 Initial Correspondence Estimation

As in other motion segmentation and registration algorithms, some initialization ofthe matching between the surfaces must be used. One approach [6] is to use nonrigidsurface matching for initialization. Another possibility, in the case of high frameraterange scanners [65], is to exploit temporal consistency by 3D tracking. Yet anotherpossible source for initial matches incorporates motion capture marker systems.Such sparse initial correspondence lends itself to interpolation of the motion field,in order to initialize a local ICP algorithm, and match the patch around each sourcepoint to the target mesh. In Fig. 12.4, we use 30 matched points for initialization.This number of points is within the scope of current motion capture marker systems,or of algorithms for global nonrigid surface matching such as spectral methods[34,41,49,52], or the generalized multidimensional scaling (GMDS) algorithm [15].

We expect that a better initial registration, as can be obtained e.g. using asmoothness assumption, or by pruning unsuitable candidates [53], will reduce thenumber of markers needed.


12.4.2 Diffusion of Lie-Group Elements

Rewriting the optimization over the functional in Eq. (12.11) in a fractional stepapproach [66], we update the parameters w.r.t. each term of the functional in asuitable representation. The treatment of regularized data fitting in a fractionalstep approach with respect to different forces has been used before for rigid bodymotion [20], and is also similar to the approach taken by Thirion’s demons algorithm[48, 58] for registration.

Using the transformation described in Sect. 12.3, the update step with respect tothe regularization now becomes

gkC1=2 D gkexp

�

�dtı�AT

ı Qg�

; vkC1 D vk � dtı�AT

ıv(12.12)

where exp.A/ D I C A C A2=2Š C A3=3Š C : : : denotes the matrix exponential, Qgdenotes the logarithm transform of g, and dt denotes the time step. ı�AT

ı Qg denotes thevariation of the regularization term �AT .g/ w.r.t. the Lie-algebra local representationof the solution, describing the Euler-Lagrange descent direction. g.x/ and theneighboring transformations are parameterized by a basis for matrices in se.3/, afterapplying the logarithm map at g.x/. The descent directions are given by

ı�AT

ı Qgi

D v2�X. Qgi / C v hrv; r Qgi i (12.13)

ı�AT

ıvD hrg; rgig.x/v C 2��X.v/ C .v � 1/

2�;

where Qgi denote the components of the logarithmic representation of g. Thediscretization we use for �X is a cotangent one suggested by [23], which has beenshown to be convergent for relatively smooth and well-parameterized surfaces. It isexpressed as

�X .u/ � 3

Ai

X

j 2N1.i/

cot ˛ij C cot ˇij

2

uj � ui

; (12.14)

for a given function u on the surface X , where N1.i/ denotes the mesh neighborsof point i , and ˛ij ; ˇij are the angles opposing the edge ij in its neighboring faces.Ai denotes the area of the one-ring around i in the mesh. After a gradient descentstep w.r.t. the diffusion term, we take a step w.r.t. the data term.

gkC1 D PSE.3/

�

gkC1=2 � dtıEDATA

ıg

�

; (12.15)


where PSE.3/.�/ denotes a projection onto the group SE.3/ obtained by correctingthe singular values of the rotation matrix [12]. We compute the gradient w.r.t. abasis for small rotation and translation matrices comprised of the regular basisfor translation and the skew-matrix approximation of small rotations. We thenreproject the update onto the manifold. This keeps the inaccuracies associated withthe projecting manifold-constrained data [19, 28, 32, 42] at a reasonable level, andleads to a first-order accuracy method. As noted by Belta and Kumar [11] in thecontext of trajectory planning and ODEs over Lie-groups, this method is reasonablyaccurate. In practice the time-step is limited in our case by the data-fitting ICP termand the explicit diffusion scheme. We expect improved handling of these terms toallow faster implementation of the proposed method.

Finally, we note that we may not know in advance the points y.x/ which matchX in Y . The correspondence can be updated based on the current transformations inan efficient manner similarly to the ICP algorithm. In our implementation we usedthe ANN library [7] for approximate nearest-neighbor search queries. We did notincorporate, however, any selective pruning of the matching candidates. These areoften used in order to robustify such the ICP algorithm against ill-suited matchesbut are beyond the scope of this chapter.

12.4.3 Visualizing Lie-Group Clustering on Surfaces

Finally, we need to mention the approach taken to visualize the transformationsas the latter belong to a six-dimensional non-Euclidean manifold. Motivatedby the widespread use of vector quantization in such visualizations, we use aclustering algorithm with spatial regularization. Instead of minimizing the Lloyd-Max quantization [33] cost function, we minimize the function

EVIS.gi ; Ri/ DX

i

Z

Ri

kg � gi k2da CZ

@Ri

v2.s/ds; (12.16)

where @Ri denotes the set of boundaries between partition regions fRigNiD1, gi are

the group representatives for each region, and v2.s/ denotes the diffusivity termalong the region boundary. The representation of members in SE.3/ is done viaits embedding into R

12, with some weight given to spatial location, by lookingat the product space R

3 � SE.3/ � R15. Several (about 50) initializations are

performed, as is often customary in clustering, with the lowest cost hypothesis kept.The visualization is detailed as Algorithm 2

While this visualization algorithm coupled with a good initialization at each pointcan be considered as a segmentation algorithm in its own right, it is less general as itassumes a strict separation between the parts. One possible question that can be raiseconcerned the meaning behind vector quantization of points belonging to a manifoldthrough its embedding into Euclidean space. In our case, since we are dealing with


Algorithm 2 Spatially-consistent clustering algorithm1: for j D 1; 2; : : : ; for a certain number of attempts do2: Use k-means on the spatial-feature space embedding, R3 � SE.3/ � R15, to get an initial

clustering.3: Use the clusters in order to optimize a spatially-regularized vector quantization measure,

C D mingi ;@Ri

Z

X

kg � gik2da CZ

@Ri

v2.s/ds;

where @Ri denotes the set of boundaries between clustered regions, gi are the transforma-tion representatives for each region, and v2.s/ denotes the diffusivity term along the regionboundary.

4: If C is lower than the lowest C found so far, keep the hypothesis.5: end for6: return current best hypothesis.

relatively well-clustered points (most of the points in a part move according to asingle transformation in SE.3/), the distances on the manifold are not large andare therefore well-approximated by Euclidean ones. We further note, however, thatthe diffusion process lowered the score obtained in Eq. (12.16) in the experimentswe conducted, indicating a consistency between the two algorithms in objects withwell-defined rigid parts.

12.5 Results

We now show the results of our method, in terms of the obtained transformationsclusters and the Ambrosio-Tortorelli diffusivity function. In Fig. 12.1 we showthe segmentation obtained by matching two human body poses taken from theTOSCA dataset [16]. We visualize the transformations obtained using the clusteringalgorithm described in Sect. 12.4.3. We initialized the transformations on the surfaceby matching the neighborhood of each surface point to the other poses using thetrue initial correspondence. The results of our method seem plausible, except forthe missing identification of the right leg, which is due to the fact that its motion islimited between the two poses.

Figure 12.1 also demonstrates the results of comparing four poses of the samesurface, this time with the patch-based data term described by (12.3). In ourexperiments the patch-based term gave a cleaner estimation of the motion, asis observed in the diffusivity function. We therefore demonstrate the results ofminimizing the functional incorporating this data term. We also show the diffusivityfunction, which hints at the location of boundaries between parts, and thus justifiesthe assumption underlying Algorithm 2.

In Figs. 12.3 and 12.2 we show the results of our algorithm on a set of six posesof a horse and camel surfaces taken from [57]. In this figure we compare our results


Fig. 12.1 Segmenting a human figure. Top row: the set of poses used. Bottom row, left to right:the transformations obtained from the two left most poses, the transformations obtained from allfour poses using Eq. (12.3) as a data term, and the Ambrosio-Tortorelli diffusivity function basedon four poses

Fig. 12.2 Segmenting a camel dynamic surface motion based on six different poses. Top row: theposes used. Bottom row, left to right: a visualization of the transformations of the surface obtainedby our method and the diffusivity function v

to those of Wuhrer and Brunton [64], obtained on a similar set of poses with tenframes. The results of our method seem to be quite comparable to those obtainedby Wuhrer and Brunton, despite the fact that we use only six poses. We also notethat both the diffusion scheme and the visualization algorithm gave a meaningfulresult for the tail part, which is not rigid and does not have a piecewise-rigid motionmodel.

In Fig. 12.4 we demonstrate our algorithm, with an initialization of 30 simulatedmotion capture marker points, where the displacement is known. The relativelymonotonous motion range available in the dynamic mesh sequence leads to a lesscomplete, but still quite meaningful, segmentation of the horse, except for its head.

We also note the relatively low number of poses required for segmentation – inboth Figs. 12.3 and 12.4 we obtain good results despite the fact that we use only afew poses, six and eight respectively.


Fig. 12.3 Segmenting a horse dynamic surface motion based on six different poses. Top row: theposes used. Bottom row, left to right: a visualization of the transformations of the surface obtainedby our method, and the segmentation results obtained by [64], and the diffusivity function v

Fig. 12.4 Segmenting a horse dynamic surface motion with a given sparse initial correspondences.Top row: the eight random poses used. Bottom row, left to right: the set of points used forinitializing the transformations, and a visualization of the transformations obtained, and thediffusivity function v

Finally, in Fig. 12.4 we demonstrate initialization of our method based on a sparsepoint set, with 30 known correspondence points. The points are arbitrarily placedusing farthest point sampling [27,31]. This demonstrates a possibility of initializingthe algorithm using motion capture markers, coupled with a 3D reconstructionpipeline, for object part analysis. While the examples shown in this chapter aresynthetic, this example shows that the algorithm can be initialized with data obtainedin a realistic setup.

12.6 Conclusion

In this chapter we present a new method for motion-based segmentation ofarticulated objects, in a variational framework. The method is based on minimizing ageneralized Ambrosio-Tortorelli functional regularizing a map from the surface onto


the Lie-group SE.3/. The results shown demonstrate the method’s effectiveness,and compare it with state-of-the-art articulated motion segmentation algorithms.The functional we suggest can be easily tailored to specific problems where it canbe contrasted and combined with domain-specific algorithms for articulated objectanalysis. In future work we intend to adapt the proposed algorithm to real datafrom range scanners, and explore initialization methods as well as use the proposedframework in other applications such as articulated surfaces tracking and denoising.

References

1. Alexander, E.J., Bregler, C., Andriacchi, T.P.: Non-rigid modeling of body segments forimproved skeletal motion estimation. Comput. Model. Eng. Sci. 4, 351–364 (2003)

2. Ambrosio, L., Tortorelli, V.M.: Approximation of functional depending on jumps by ellipticfunctional via � -convergence. Commun. Pure Appl. Math. 43(8), 999–1036 (1990)

3. Amiaz, T., Kiryati, N.: Piecewise-smooth dense optical flow via level sets. Int. J. Comput. Vis.68(2), 111–124 (2006)

4. Andersen, M.S., Benoit, D.L., Damsgaarda, M., Ramsey, D.K., Rasmussen, J.: Do kinematicmodels reduce the effects of soft tissue artefacts in skin marker-based motion analysis? An invivo study of knee kinematics. J. Biomech. 43, 268–273 (2010)

5. Anguelov, D., Koller, D., Pang, H.-C., Srinivasan, P., Thrun, S.: Recovering articulated objectmodels from 3D range data. In: Proceedings of the Conference on Uncertainty in ArtificialIntelligence, pp. 18–26. AUAI Press, Arlington (2004)

6. Arcila, R., Kartik Buddha, S., Hetroy, F., Denis, F., Dupont, F.: A framework for motion-based mesh sequence segmentation. In: International Conference on Computer Graphics,Visualisation and Compter Vision, Plzen, Czech Republic, 2010

7. Arya, S., Mount, D.M., Netanyahu, N.S., Silverman, R., Wu, A.Y.: An optimal algorithm forapproximate nearest neighbor searching fixed dimensions. J. ACM 45(6), 891–923 (1998)

8. Ashbrook, A.P., Fisher, R.B., Robertson, C., Werghi, N.: Segmentation of range data into rigidsubsets using surface patches. In International Conference on Computer Vision, pp. 201–206.IEEE Computer Society, Washington, DC (1998)

9. Attene, M., Katz, S., Mortara, M., Patane, G., Spagnuolo, M., Tal, A.: Mesh segmentation – acomparative study. In: Proceedings of the IEEE International Conference on Shape Modelingand Applications, pp. 7–18. IEEE Computer Society, Washington, DC (2006)

10. Belkin, M., Sun, J., Wang, Y.: Constructing Laplace operator from point clouds in Rd . In:

Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA’09, pp. 1031–1040. Society for Industrial and Applied Mathematics, Philadelphia (2009)

11. Belta, C., Kumar, V.: On the computation of rigid body motion. Electron. J. Comput. Kinemat.1 (2001)

12. Belta, C., Kumar, V.: An SVD-based project method for interpolation on se(3). IEEE Trans.Robot. Autom. 18(3), 334–345 (2002)

13. Besl, P.J., McKay, N.D.: A method for registration of 3D shapes. IEEE Trans. Pattern Anal.Mach. Intell. 14(2), 239–256 (1992)

14. Brockett, R.: Robotic manipulators and the product of exponential formula. In: Fuhrman,P. (ed.) Proceedings of the Mathematical Theory of Networks and Systems, pp. 120–129.Springer (1984)

15. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Generalized multidimensional scaling: aframework for isometry-invariant partial surface matching. Proc. Natl. Acad. Sci. USA 103(5),1168–1172 (2006)

16. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical geometry of non-rigid shapes.Springer, New York (2008)


17. Bruhn, A., Weickert, J., Schnorr, C.: Lucas/Kanade meets Horn/Schunck: combining local andglobal optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)

18. Brune, C., Maurer, H., Wagner, M.: Detection of intensity and motion edges within optical flowvia multidimensional control. SIAM J. Imag. Sci. 2(4), 1190–1210 (2009)

19. Celledoni, E., Owren, B.: Lie group methods for rigid body dynamics and time integration onmanifolds. Comput. Methods Appl. Mech. Eng. 19, 421–438 (1999)

20. Celledoni, E., Fasso, F., Safstrom, N., Zanna, A.: The exact computation of the free rigid bodymotion and its use in splitting methods. SIAM J. Sci. Comput. 30(4), 2084–2112 (2008)

21. Chen, X., Golovinskiy, A., Funkhouser, T.: A benchmark for 3D mesh segmentation. ACMTrans. Graph. 28(3), Article 73 (2009)

22. Daniel Cremers and Steffano Soatto: Motion competition: a variational framework for piece-wise parametric motion segmentation. Int. J. Comput. Vis. 62(3), 249–265 (2005)

23. Desbrun, M., Meyer, M., Schroder, P., Barr, A.H.: Implicit fairing of irregular meshesusing diffusion and curvature flow. SIGGRAPH Proceedings of the 26th Annual Conferenceon Computer Graphics and Interactive Techniques, pp. 317–324. ACM/Addison-Wesley,New York (1999)

24. do Carmo, M.P.: Riemannian Geometry. Birkhauser Verlag, Boston (1992)25. Duits, R., Burgeth, B.: Scale spaces on Lie groups. In: Scale Space and Variational Methods in

Computer Vision. LNCS, vol. 4485, pp. 300–312. Springer, Berlin/Heidelberg (2007)26. Eells, J. Jr., Sampson, J.H.: Harmonic mappings of Riemannian manifolds. Am. J. Math. 86(1),

106–160 (1964)27. Gonzalez T.F.: Clustering to minimize the maximum intercluster distance. Theor. Comput. Sci.

38, 293–306 (1985)28. Gur, Y., Sochen, N.A.: Regularizing flows over lie groups. J. Math. Imag. Vis. 33(2), 195–208

(2009)29. Hall, B.C.: Lie Groups, Lie Algebras,and Representations, An Elementary Introduction.

Springer-Verlag, 2004.30. Hauberg, S., Sommer, S., Pedersen, K.S.: Gaussian-like spatial priors for articulated tracking.

In: European Conference on Computer Vision. LNCS, vol. 6311, pp. 425–437. Springer,Berlin/New York (2010)

31. Hochbaum, D., Shmoys, D.: A best possible approximation for the k-center problem. Math.Oper. Res. 10(2), 180–184 (1985)

32. Iserles, A., Munthe-Kaas, H.Z., Nørsett, S.P., Zanna, A.: Lie group methods. Acta Numer. 9,215–365 (2000)

33. Jain, A.K.: Fundamentals of Digital Image Processing. Prentice-Hall, Upper Saddle River(1989)

34. Jain, V., Zhang, H.: Robust 3D shape correspondence in the spectral domain. In: Proceedings ofShape Modeling International, pp. 118–129. IEEE Computer Society, Washington DC (2006)

35. James, D.L., Twigg, C.D.: Skinning mesh animations. SIGGRAPH 24(3), 399–407 (2005)36. Kalogerakis, E., Hertzmann, A., Singh, K.: Learning 3D mesh segmentation and labeling.

ACM Trans. Graph. 29(4), Article 102 (2010)37. Kobilarov, M., Crane, K., Desbrun, M.: Lie group integrators for animation and control of

vehicles. ACM Trans. Graph. 28(2), 1–14 (2009)38. Kompatsiaris, I., Tzovaras, D., Strintzis, M.G.: Object articulation based on local 3D motion

estimation. In: Proceedings of ECMAST, pp. 378–391. Springer, London (1999)39. Lee, T.-Y., Wang, Y.-S., Chen, T.-G.: Segmenting a deforming mesh into near-rigid compo-

nents. Vis. Comput. 22(9), 729–739 (2006)40. Litke, N., Droske, M., Rumpf, M., Schroder, P.: An image processing approach to sur-

face matching. In: Proceedings of the Eurographics Symposium on Geometry processing,pp. 207–216. Eurographics Association, Aire-la-Ville (2005)

41. Mateus, D., Horaud, R., Knossow, D., Cuzzolin, F., Boyer, E.: Articulated shape matchingusing Laplacian eigenfunctions and unsupervised point registration. In: CVPR. IEEE ComputerSociety (2008)

42. Muller, A., Terze, Z.: Differential-geometric modelling and dynamic simulation of multibodysystems. Strojarstvo 51(6), 597–612 (2009)


43. Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associatedvariational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)

44. Murray, R.M., Li, Z., Sastry, S.S.: A Mathematical Introduction to Robotic Manipulation,1 edn. CRC, Boca Raton (1994)

45. Nir, T., Kimmel, R., Bruckstein, A.M.: Variational approach for joint optic-flow computationand video restoration. Technical Report CIS-05, Technion, 2005

46. Nir, T., Bruckstein, A.M., Kimmel, R.: Over-parameterized variational optical flow. Int.J. Comput. Vis. 76(2), 205–216 (2008)

47. Park, F.C., Bobrow, J.E., Ploen, S.R.: A Lie group formulation of robot dynamics. Int. J. Robot.Res. 14, 609–618 (1995)

48. Pennec, X., Cachier, P., Ayache, N.: Understanding the “demon’s algorithm”: 3D non-rigid registration by gradient descent. In: MICCAI Proceedings of the Second InternationalConference on Medical Image Computing and Computer-Assisted Intervention, pp. 597–605.Springer, London (1999)

49. Raviv, D., Dubrovina, A., Kimmel, R.: Hierarchical shape matching. In: Bruckstein, A.M.,ter Haar Romeny, B., Bronstein, A.M., Bronstein, M.M. (eds.) Scale Space and VariationalMethods in Computer Vision. LNCS, vol. 6667, pp. 604–615 (2011)

50. Rosman, G., Bronstein, M.M., Bronstein, A.M., Wolf, A., Kimmel, R.: Group-valued regular-ization framework for motion segmentation of dynamic non-rigid shapes. In: Scale Space andVariational Methods in Computer Vision, pp. 725–736 (2011)

51. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms.Phys. D Lett. 60, 259–268 (1992)

52. Ruggeri, M.R., Patane, G., Spagnuolo, M., Saupe, D.: Spectral-driven isometry-invariantmatching of 3D shapes. Int. J. Comput. Vis. 89(2–3), 248–265 (2010)

53. Szymon Rusinkiewicz and Marc Levoy: Efficient variants of the ICP algorithm. In: ThirdInternational Conference on 3D Digital Imaging and Modeling (3DIM), pp. 145–152. IEEEComputer Society (2001)

54. Sapp, B., Toshev, A., Taskar, B.: Cascaded models for articulated pose estimation. In: EuropeanConference Computer Vision, pp. 406–420. Springer, Berlin/New York (2010)

55. Shamir, A.: A survey on mesh segmentation techniques. Comput. Graph. Forum 27(6), 1539–1556 (2008)

56. Subbarao, R., Meer, P.: Nonlinear mean shift over Riemannian manifolds. Int. J Comput. Vis.84(1), 1–20 (2009)

57. Sumner, R.W., Popovic, J.: Deformation transfer for triangle meshes. In: SIGGRAPH,pp. 399–405. ACM, New York (2004)

58. Thirion, J.P.: Image matching as a diffusion process: an analogy with Maxwell’s demons. Med.Image Anal. 2(3), 243–260 (1998)

59. Tierny, J., Vandeborre, J.-P., Daoudi, M.: Fast and precise kinematic skeleton extraction of 3Ddynamic meshes. In: ICPR. IEEE Computer Society (2008)

60. Trobin, W., Pock, T., Cremers, D., Bischof, H.: An Unbiased Second-Order Prior for High-Accuracy Motion Estimation, pp. 396–405. Springer, Berlin/New York (2008)

61. Tuzel, O., Porikli, F., Meer, P.: Learning on lie groups for invariant detection and tracking. In:CVPR. IEEE Computer Society (2008)

62. Wang, S., Wang, Y., Jin, M., Gu, X.D., Samaras, D.: Conformal geometry and its applicationson 3D shape matching, recognition, and stitching. IEEE Trans. Pattern Anal. Mach. Intell.29(7), 1209–1220 (2007)

63. Wolf, A., Sharf, I., Rubin, M.B.: Using Cosserat point theory for estimating kinematics andsoft-tissue deformation during gait analysis. In: Lenarcic, J., Stanisic, M.M. (eds.) Advancesin Robot Kinematics: Motion in Man and Machine, pp. 63–70. Springer, Netherlands (2010)

64. Wuhrer, S., Brunton, A.: Segmenting animated objects into near-rigid components. Vis.Comput. 26, 147–155 (2010)

65. Yamasaki, T., Aizawa, K.: Motion segmentation for time-varying mesh sequences based onspherical registration. EURASIP J. Appl. Signal Process. 1–9, (2009). Article 346425


66. Yanenko, N.N.: The Method of Fractional Steps: Solution of Problems of MathematicalPhysics in Several Variables. Springer, Berlin/New York (1971). Translated from Russian

67. Yang, C., Medioni, G.: Object modelling by registration of multiple range images. Image Vis.Comput. 10, 145–155 (1992)

68. Zhang, D., Hebert, M.: Harmonic maps and their applications in surface matching. In: CVPR,vol. 2, pp. 524–530. IEEE Computer Society (1999)

69. Zefran, M., Kumar, V., Croke, C.: On the generation of smooth three-dimensional rigid bodymotions. IEEE Trans. Robot. Autom. 14(4), 576–589 (1998)

70. Zefran, M., Kumar, V., Croke, C.: Metrics and connections for rigid-body kinematics. Int.J. Robot. Res. 18(2), 242 (1999)

Documents

[Mathematics and Visualization] Innovations for Shape Analysis || Group-Valued Regularization for Motion Segmentation of Articulated Shapes