Coherent Laplacian 3D protrusion segmentation Oxford Brookes Vision Group Queen Mary, University of...

Coherent Laplacian 3D protrusion segmentation

Oxford Brookes Vision Group

Queen Mary, University of London, 11/12/2009

Fabio Cuzzolin

The problem

to recognize actions we need to extract features

segmenting moving articulated 3D bodies into parts

along sequences, in a consistent way

in an unsupervised fashion

robustly, with respect to changes of the topology of the moving body

as a building block of a wider motion analysis and capture framework

ICCV-HM'07, CVPR'08, to submit to IJCV

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Spectral methods

given a dataset of points {Xi, i=1,..,N}

compute an affinity matrix A(i,j) = d(Xi,Xj)

apply SVD to this affinity matrix

this yields a list of eigenvalues and associated eigenvectors

a number of eigenvectors are selected, and used to build an “embedded cloud” of points {Yi, i=1,..,N}

Laplacian methods

the affinity matrix is the Laplacian operator, or some function of it

Graph Laplacian: operator on functions f defined on sets X of points of the form

L[f]i = jN(i) wij (fi fj)

maps each such function f to another function L[f]

N(i) is the set of neighbors of Xi

fi is the value of the function f on Xi

Laplacian eigenfunctions

Laplacian eigenfunctions/values have nice topological properties

eigenvalues are invariant for volume-preserving transformations

eigenfunctions for a “base” for all functions on X

their zero-level sets are related to protrusions and symmetries of the underlying cloud

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Locally Linear Embedding

• for each data point we compute the weights Wij that best reconstruct Xi from its neighbors:

argminW i |X

i

j W

ij X

j|2

Low-dim embeddings Y_i are obtained by

argminY i |Y

i

j W

ij Y

j|2

i.e., local neighbors are the same, subject to • affinity matrix M = (IW)T(IW)

optimal embedding → bottom d+1 eigevectors (but last one)

LLE algorithm

Protrusion preservation

protrusions are high-curvature surface regions

by definition LLE leaves unchanged the weights of each neighborhood (the affine coordinates of X_i in the base of its neighbors)

weights depend on pairwise distances between points

preserving weights means preserving distances up to a scale

this happens in surface neighborhoods too

if they are all roughly the same size ...

... curvature distribution is preserved: protrusions!

Lower dimensionality

protrusion preservation is an effect of local isometry (the first constraint of LLE)

the covariance constraint has the effect of producing a lower-dimensional embedded cloud

LLE is a constrained minimization problem

in physics constraints G(X)=0 are associated with a force orthogonal to the constraint surface

the covariance constraint is associated with a force that pulls the cloud of points outward, reducing the chain of neighborhoods to a “string”

Clustering in the embedding space

Locally Linear Embedding: preserves the local Locally Linear Embedding: preserves the local structure of the datasetstructure of the dataset

generates a lower-dim embedded cloudpreserves protrusionsless sensitive to topology changes than other methods

40 50 60 70

60

8010

20

30

40

50

60

70

80

3D shape

0.05

0

0.05

0.060.04

0.020

0.02

0.05

0

0.05

0.1

LLE space

50

40

30

20

10

010

20

30

40

40

30

20

10

0

10

20

30

40

50

ISOMAP space

Pose invariance with LLE

generates a lower-dim, widely separated embedded cloudless sensitive to topology changes than other methodsless expensive then ISOMAP (refs. Jenkins, Chellappa)

rigid part

rigid part

moving joint area

unaffected neighborhoods

unaffected neighborhoods

affected neighborhoods

local neighbourhoods stable under articulated motion

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Algorithm

due to their low dimensionality, protrusions are detected in the embedding spacethey can be clustered as sets of collinear points using k-wise clusteringsegmentation is brought back to 3D

K-wise clustering

LLE maps the 3D shape to a lower-dimensional shapeIdea: clustering collinear points together

• K-wise clustering:K-wise clustering:

a hypergraph H is built by measuring the affinity of all triadsa weighted graph G which approximates H is constructed by constrained linear least square optimizationthe approximating graph is partitioned by spectral clustering (n-cut)

Protrusion detection

protrusions can be easily detected after embeddingdue to low dimensionality

Branch termination not detected Branch termination detected

an embedded point is a termination if its projection on the line interpolating its neighborhood is an extremum

Seed propagation along time

To ensure time consistency clusters’ seeds have to be propagated along time

Old positions of clusters in 3D are added to new cloud and embedded

Result: new seeds

Merging/splitting clusters

At each t all branch terminations of Y(t) are detected;

if t=0 they are used as seeds for k-wise clustering;

otherwise (t>0) standard k-means is performed on Y(t) using branch terminations as seeds, yielding a rough partition of the embedded cloud into distinct branches;

propagated seeds in the same partition are merged;

for each partition of Y(t) not containing any old seed a new seed is defined as the related branch termination.

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Results on real sequences 1

for real sequences ground truth is difficult to gather

we can still visually appreciate the quality and consistency of the resulting segmentation

Results on real sequences 2

for real sequences ground truth is difficult to gather

we can still visually appreciate the quality and consistency of the resulting segmentation

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Ground truth for synthetic data

For synthetic sequences of sequences for which pose has been estimated, ground truth can be gathered

performance indicators: compare the obtained segmentation with the three “natural” ones on the right

Scores for synthetic sequences

Scores for synthetic sequences

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Handling topology changes

when topology changes occur cluters merge and/or split to accommodate them

Handling missing data

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Vs EM clustering

EM clustering fits a multi-Gaussian distribution to the data through the EM algorithmnumber of cluster is automatically estimatedLeft: our algo; right: EM clustering

Vs ISOMAP

the same propagation scheme can be applied in the ISOMAP space

• extremely sensitive to topology changes

ISOMAP computes an embedding by applying MDS to the affinity matrix of all pairwise geodesic distances

Performance comparison

segmentation scores for two other real sequencessolid: our method; dashed: EM; dashdot: ISOMAP

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Estimating neighborhood size

the number of neighbors can be estimated from the data sequenceadmissible k: yields neighborhoods which do not span different bodyparts

0 20 40 60 80 1000

1

2

3

4

5

6

7

05

1015

20

05

1015

0

5

10

15

20

25

30

35

40

45

50

0

10

20

0

10

0

5

10

15

20

25

30

35

40

45

50

0 5 10 15 20 25 300

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Eigenvector selection

according to the eigenvectors we select after decomposition, we obtain different unsupervised segmentations

Stability with respect to parameter values

• consistency, segmentation and average scores for 2 sequences, as a function of parameter values k and d

Coherent 3D Laplacian protrusion segmentation

Laplacian methods

Locally linear embedding

An unsupervised algorithm

Results on real sequences

Results on synthetic sequences

Topology changes and missing data

Comparisons

Influence of parameters

Applications

Model recovery example

a sequence representing a counting hand is portrait

along time, the algorithm “learns” the object is formed by more and more rigid segments

Laplacian matching of dense meshes or voxelsets

as embeddings are pose-invariant (for articulated bodies)

they can then be used to match dense shapes by simply aligning their images after embedding

ICCV '07 – NTRL, ICCV '07 – 3dRR, CVPR '08, to submit to PAMI

Eigenfunction Histogram assignment

Algorithm:

compute Laplacian embedding of the two shapesfind assignment between eigenfunctions of the two shapesthis selects a section of the embedding spaceembeddings are orthogonally aligned there by EM

Results

Appls: graph matching, protein analysis, motion capture To propagate bodypart segmentation in timeMotion field estimation, action segmentation

Conclusions

Unsupervised bodypart segmentation algorithm which ensure consistency along timeSpectral method: clustering is performed in the embedding space (in particular after LLE) as shape becomes lower-dim and different bodyparts are widely separatedSeeds are propagated along time and merged/splitted according to topology variationsCompares favorably with other techniquesFirst block of motion analysis framework (matching, action recognition, etc.)