Team 5: Size and Shape Comparison · Problem Formulation Coordinates-based Methods Invariant Methods Team 5: Size and Shape Comparison Mentor: Mark A. Stuﬀ Team Members: Suman Balasubramanian1

Problem FormulationCoordinates-based Methods

Invariant Methods

Team 5: Size and Shape Comparison

Mentor: Mark A. Stuff

Team Members:Suman Balasubramanian1

Ying Wai (Daniel) Fan2

Yuen Yick Kwan3

Josef Aaron Sifuentes4

Jinglong Ye1

Weifeng Zhi5

1Mississippi State U., 2Emory U., 3Purdue U., 4Rice U., 5U. Kentucky

Team 5: Size and Shape Comparison


Invariant Methods

Outline

Problem Formulation

Coordinates-based MethodsRigid Procrustes AnalysisIterative Closest Point (ICP) Algorithm

Invariant MethodsComparing Lists of Geometric InvariantsComparing Matrices of Geometric InvariantsSubset Selection Function Estimation

When m = n − 1When m < n − 1

2 / 41


Invariant Methods

Problem Motivation

I The availability, usage and mobility of sensor components inrecording objects of interest enable generation of many formsof three dimensional data.

I In order to find the correspondence between the threedimensional data sets when no prior knowledge is available,new methods of estimating the transformation from a set ofunlabeled landmarks are required.

3 / 41


Invariant Methods

Optimization Problem

Given two sets M and N with |M| < |N|, find S , Π, R and T suchthat

‖M − SΠNR − T‖F

is minimized, where

S = selection matrix,

Π = permutation matrix,

R = rotation matrix,

T = translation.

4 / 41


Invariant Methods

Rigid Procrustes AnalysisIterative Closest Point (ICP) Algorithm

Outline

Problem Formulation




5 / 41


Invariant Methods


Outline

Problem Formulation




6 / 41


Invariant Methods


Rigid Procrustes Analysis

Suppose |M| = |N| = n and no permutation, then we have tominimize

minR,T

‖N −MR − T‖F . (1)

The result from Procrustes analysis gives

R = UV T , (2)

where N̂T M̂ = VΛUT with M̂ and N̂ are centered M and N,U,V ∈ SO(3) and Λ = diag(λ1, λ2, . . . , λn) withλ1 ≥ λ2 ≥ · · ·λn−1 ≥ |λn|;

T = cN − cMR, (3)

where cM and cN are the centroids of points in M and Nrespectively.

7 / 41


Invariant Methods


Outline

Problem Formulation




8 / 41


Invariant Methods


Iterative Closest Point (ICP)

I Iteratively estimates the rigid motion between M and N.I Steps:

0 Find initial guesses of rigid motion:I Find a large tetrahedron in M and some tetrahedrons in N

that look like this tetrahedron.I The initial guesses are the rigid motions between the

tetrahedrons.

1 Associate points in M to points in N using closest neighbor.2 Recompute the rigid motion (e.g. Procrustes).3 Iterate (go back to 1) until convergence.

Source: Rusinkiewicz and Levoy “Efficient Variants of the ICP Algorithm”, Third International Conference on 3-D

Digital Imaging and Modeling, 2001

9 / 41


Invariant Methods


ICP Results

Left: Original cow.Middle: Perturbed cow (blue) and the recovered cow (red) withσ = 0.01.Right: Perturbed cow (blue) and the recovered cow (red) withσ = 0.01.

10 / 41


Invariant Methods

Comparing Lists of Geometric InvariantsComparing Matrices of Geometric InvariantsSubset Selection Function Estimation

Outline

Problem Formulation




11 / 41


Invariant Methods


Outline

Problem Formulation




12 / 41


Invariant Methods


Examples of geometric invariants

I Sets of distances from the centroid to each vertex: n

invariants

I Set of all pairwise distances between the points,(n2

)invariants.

I M. Boutin, G. Kemper,On reconstructing n-pointconfigurations from the distribution of distances or areas,Advances in Applied Mathematics,2(2004)709− 735.

I Set of areas of all triangles:(n3

)invariants.

I · · ·

13 / 41


Invariant Methods


Outline

Problem Formulation




14 / 41


Invariant Methods


Inner Products Invariants

Compile the two sets of n points {mi}ni=1 and {ni}n

i=1 intomatrices

M = [m1 m2 · · · mn]T ∈ Rn×3

N = [n1 n2 · · · nn]T ∈ Rn×3

Then if we build the matrices of inner product relations

FM = MMT

FN = NNT

The matrices are the same under permutation.Note also that these matrices are rank 3.

15 / 41


Invariant Methods


Spectral Comparisons of Inner Product Matrices

Thus if Π is the permutation matrix that corresponds like pointsunder the rigid motion

ΠFMΠT = FN ,

which indicates that FM and FN are similar. Thus

σ(FM) = σ(FN)

We also obtain that σ(MTM) = σ(NTN)Then we construct a score function SI for similarity

SI =3∑

i=1

|λNi − λM

i |2

This gives a way to compare likeness of two sets of points under arigid motion and permutation without knowing what the rigidmotion or permutation is.

16 / 41


Invariant Methods


Comparisons of Distance Matrices

Similarly if we build the matrix of distance relations

[DM ]ij = ‖mi −mj‖[DN ]ij = ‖ni − nj‖

As in the case of the inner product matrices, the two distancematrices are the same under permutation. Therefore

σ(DM) = σ(DN)

We construct another score function SD for similarity

SD =1

n

n∑i=1

|λNi − λM

i |2

Thus we have another way of comparing likeness of two sets ofpoints under a rigid motion and permutation without knowingwhat the rigid motion or permutation is.

17 / 41


Invariant Methods


Subset selection by eigenvalue comparison.

I Method requires searching(nm

)possible subsets.

I Matching eigenvalues does not guarantee isometric subsets.

18 / 41


Invariant Methods


Lyapunov Equations

Thus we can solve for the permutation by solving

ΠDM −DNΠ = 0

These are known as the Lyapunov Equations (also called theSylvester Equations). We can solve these by vectorizing the aboveequations

vec(ΠDM)− vec(DNΠ) = 0

⇐⇒ [(DM ⊗ I)− (I⊗DN)] ~Π = 0

where ~Π is the vectorized permutation matrix.

19 / 41


Invariant Methods


Solving Lyapunov Equations

This implies that

~Π ∈ N ( (DM ⊗ I)− (I⊗DN) )

I Let (λMi , vM

i )ni=1 be the eigenpairs of DM

I and (λNi , vN

i )ni=1 be the eigenpairs of DN

I then ( (λMi − λN

j ), (vMi ⊗ vN

j ) )ni ,j=1 are eigenpairs of( (DM ⊗ I)− (I⊗DN) ).

Then if M and N are the same under a rigid motion

~Π ∈ span( (vM1 ⊗ vN

1 ), (vM2 ⊗ vN

2 ), . . . , (vMn ⊗ vN

n ) ).

20 / 41


Invariant Methods


Solving Lyapunov Equations

Let V be the matrix whose vectors are the above basis of our nullspace, then there exists α ∈ Rn s.t.

Vα = ~Π

We add the constraints that the row sum and column sum of Π isone by enforcing the constraints

(eT ⊗ I)~Π = e

(I⊗ eT )~Π = e

where e is the vector of all ones in Rn.

21 / 41


Invariant Methods


Solving Lyapunov Equations-two strategies

One strategy is [(eT ⊗ I)~Π

(I⊗ eT )~Π

]Vα =

[ee

]I Cheaper in terms of computer memoryI More susceptible to noise in the data.

Our second strategy is DM ⊗ I− I⊗DN

ω(eT ⊗ I)~Π

ω(I⊗ eT )~Π

~Π = ω

0ee

where ω is a weight we use to enforce the column and row sumconditions more strongly than the null space condition.

I More expensive in terms of computer memoryI More robust to noisy data.

22 / 41


Invariant Methods


Examples

Figure: Given two sets of cows in R3, where the blue cow is a rigidmotion and permutation of the red cow.

23 / 41


Invariant Methods


Examples

Figure: By solving for the permutation, we can use Procrustes to obtainthe optimal rigid motion to line up the cows.

24 / 41


Invariant Methods


Examples

Figure: By using the same algorithm we get robust results for noise withstandard deviation of σ = 0.01

25 / 41


Invariant Methods


Examples

Figure: By using the same algorithm we get robust results for noise withstandard deviation of σ = 0.1

26 / 41


Invariant Methods


Outline

Problem Formulation




27 / 41


Invariant Methods


I Objective: to find the points in N that are also in MI without computing the rigid motion and permutationI without trying all n!

m!(n−m)!) m-subsets of N.

I We denote the distance between ni and nj by di ,j .

I Suppose DN and DM are the set of pairwise distances ofpoints in N and M respectively.

I Fix some ordering in DN , say

DN ={

d1, d2, . . . , dl , . . . , d n(n−1)2

}= {d1,2, d1,3, . . . , di ,j , . . . , dn−1,n} , 1 ≤ i < j ≤ n.

28 / 41


Invariant Methods


I Define the edge selection vector s as

sl =

{1, if dl ∈ DM

0, otherwise(4)

I Let f1, . . . , fp be some linearly independent functions. Formthe function value matrix F with entries

Fk,l = fk(dl). (5)

I Then if b = F s, then

bk =

n(n−1)2∑

l=1

Fk,lsl =

n(n−1)2∑

l=1

fk(dl)sl =∑

dl∈DM

fk(dl) (6)

29 / 41


Invariant Methods


I Therefore, given N and M, we can compute the DN and DM

and from which we can compute F and b by (5) and (6) andthen solve for s in

F s = b (7)

to know the membership of dl ’s in DM and thus also thewhich ni ’s is in M.

30 / 41


Invariant Methods


Our choice of fk ’s

I In our experiments, we use the family of cosine functions asfk ’s:

fk(d) = cos(kπd).

Entries of F are bounded between −1 and 1.

I The number of rows, p, of F has to be at least n(n−1)2 . To

make our linear system better conditioned, we may set p to belarger than n(n−1)

2 and instead solve the least square problem:

mins‖b− F s‖2

F (8)

31 / 41


Invariant Methods


Outline

Problem Formulation




32 / 41


Invariant Methods


When m = n − 1

I Construct the edge matrix E :

El ,j =

{1, if nj is an endpoint of the edge dl

0, otherwise

I The point selection vector x be defined as

xj =

{1, if nj ∈ M

−1, otherwise,

I Then given M and N, we can compute F , E and b and thensolve for x in

FEx = F (2s) = 2F s = 2b. (9)

to know which point in N is in M.

33 / 41


Invariant Methods


I Note that FE is of size p by n while F is of size p by n(n−1)2 ,

so (9) is easier to solve than (7) and it saves the step offinding points connecting to edges selected by s.

Figure: F matrix Figure: FE matrix

34 / 41


Invariant Methods


Outline

Problem Formulation




35 / 41


Invariant Methods


When m < n − 1

I When M is missing more than one point of N, (9) is only anapproximation, i.e.

FEx ≈ 2b, (10)

I We do not obtain from solving (9) a x with entries -1 and 1,but some perturbed version of it.

I However, from our experiments, when we choose the numberof rows, p, in F and b large enough, say p = n2, theperturbation in x is not large enough to change the signs.

I So after taking the sign operation to the computed x, we stillget a vector of entries -1 and 1.

36 / 41


Invariant Methods


Subset Algorithm Pictures

Figure: We are given two clouds of random points, where the blue cloudis a subset of the red cloud under a permutation and rigid motion. Aboveare the respective distance matrices.

37 / 41


Invariant Methods



Figure: By using our subset selection algorithm, we can pick which redpoints correspond to the blue points under a rigid motion andpermutation. By taking out the appropriate rows and columns of ourdistance matrix associated to the red points, we get two equal matricesunder permutation. 38 / 41


Invariant Methods



Figure: By using our permutation selection algorithm, we can determinethe point correspondence in order to apply Procrustes. Notice that nowthe distance matrices are the same, as they are invariant under a rigidmotion.

39 / 41


Invariant Methods



Figure: Applying Procrustes now gives us the correct rigid motion tocorrespond our two clouds of points.

40 / 41


Invariant Methods


Thank You for Your Attention and Hospitality.Any Questions?

41 / 41

Documents

Team 5: Size and Shape Comparison · Problem Formulation Coordinates-based Methods Invariant Methods Team 5: Size and Shape Comparison Mentor: Mark A. Stuﬀ Team Members: Suman Balasubramanian1