1 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Feature-based methods and shape retrieval problems © Alexander

1Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems

Feature-based methodsand shape retrieval problems

© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book

048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes

EE Technion, Spring 2010


Structure

Local

Feature descriptors

Global

Metric


Combining local and global structures

BBK 2008; Keriven, Torstensen 2009; Dubrovina, Kimmel 2010; Wang, B 2010

Pair-wise stress (global) Point-wise stress (local)

Local structure can be geometric or photometric


Photometric stress

Thorstenstein & Keriven 2009

5Numerical Geometry of Non-Rigid Shapes Diffusion Geometry

Heat kernels, encore

Brownian motion on X starting at point x, measurable set C

probability of the Brownian motion to be in C at time t

Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005

Heat kernel represents transition probability


Intrinsic descriptors

Sun, Ovsjanikov & Guibas 2009

Multiscale local shape descriptor (Heat kernel signature)

can be interpreted as probability of Brownian motion to return to

the same point after time (represents “stability” of the point)

Time (scale)


Sun, Ovsjanikov & Guibas 2009for small t

Relation to curvature


Heat kernel signature

Heat kernel signatures represented in RGB space

Sun, Ovsjanikov & Guibas 2009Ovsjanikov, BB & Guibas 2009


Heat kernel descriptors

Invariant to isometric deformations Localized sensitivity

to topological noise

J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009M. Ovsjanikov, BB, L. Guibas, 2009


Scale invariance

Original shape Scaled by

HKS= HKS=

Not scale invariant!


Scale-invariant heat kernel signature

B, Kokkinos CVPR 2010

Log scale-space

Scaling = shift and multiplicative

constant in HKS

log + d/d

Undo scaling

Fourier transformmagnitude

Undo shift

0 100 200 300-15

-10

-5

0

t0 100 200 300

-0.04

-0.03

-0.02

-0.01

0

t0 2 4 6 8 10 12 14 16 18 20

0

1

2

3

4

=2k/T


Scale invariance

B, Kokkinos 2009

Heat Kernel Signature Scale-invariantHeat Kernel Signature


Bending invariance

B, Kokkinos CVPR 2010

Heat Kernel Signature Scale-invariantHeat Kernel Signature


Bending invariance

Wang, B 2010

Geodesic+HKS Diffusion+HKS Commute+SI-HKS


Topology invariance

Geodesic+HKS Diffusion+HKS

Wang, B 2010


Scale invariance

Wang, B 2010

Geodesic+HKS Commute+SI-HKS


Invariance

Geodesic metric

Rigid Inelastic Topology

Diffusion metric

Scale

Wang, B 2010

Commute-timemetric

Heat kernelsignature (HKS)

Scale-invariant HKS (SI-HKS)



Tagged shapes

Shapes withoutmetadata

Man, person, humanPersonText search

Content-based search

3D warehouse


?

Content-based search problems

Invariant shape retrievalShape classification

?

Semantic

Variability of shape

within category

Geometric

Variability of shape

under transformation


Image vs shape retrieval

Illumination View Missing data

Deformation Topology Missing data


Bags of words

Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period.

con

stru

ctio

nar

chit

ectu

reIt

aly

Fra

nce

cath

edra

lch

urc

hb

asili

caP

aris

Ro

me

Go

thic

Ro

man

St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy.

Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period.

St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy.


Bags of features

Visual vocabulary

Feature detector + descriptor

Invariant to changes of the image

Discriminative (tells different images apart)


Advantages

“Shape signature”

Easy to store

Easy to compare

Partial similarity possible


Images vs shapes

Images Shapes

Many prominent features Few prominent features

Affine transforms, illumination,

occlusions, resolution

Non-rigid deformations, topology,

missing parts, triangulation

SIFT, SURF, MSER, DAISY, … Curvature, conformal factor,

local distance histograms


“ShapeGoogle”

Feature descriptor

Geometric words

Bag of words

Geometric expressions

Spatially-sensitive bag of features

“ ”

“ ”


Geometric vocabulary

M. Ovsjanikov, BB, L. Guibas, 2009


Bags of features



Nearest neighbor in the descriptor space


Bags of features



Weighted distance to words

in the vocabulary


Bags of features

Shape distance = distance between bags of features


Statistics of different geometric words over the entire shape


Index in vocabulary1 64


Bags of features

32Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Statistical weighting

Query q Database D

syzygy in astronomy means alignment of three bodies of the solar system along a straight or nearly straight line. a planet is in syzygy with the earth and sun when it is in opposition or conjunction. the moon is in syzygy with the earth and sun when it is new or full.

syzygy in astronomy means alignment of three bodies of the solar system along a straight or nearly straight line. a planet is in syzygy with the earth and sun when it is in opposition or conjunction. the moon is in syzygy with the earth and sun when it is new or full.

Sivic & Zisserman 2003BB, Carmon & Kimmel 2009

Frequent in document = important

in is

or

syzygy

Rare in database = discriminative

with

a

of

the

and when

33Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Statistical weighting

Query q Database D

Significance of a term t

Term frequency Inverse documentfrequency

Weight bags of features by tf-idf

Reduce the influence of non-important terms in dense descriptor

Sivic & Zisserman 2003BB, Carmon & Kimmel 2009


Expressions

In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death.

Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population.

mat

rix

dec

om

po

siti

on

mat

rix

fact

ori

zati

on

scie

nce

fic

tio

nca

no

nic

al f

orm


In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death.


mat

rix

dec

om

po

siti

on is a

the of in to by

scie

nce

form





Expressions


mat

rix

dec

om

po

siti

on is a

the of in to by

scie

nce

form

In particular matrix used type a some science, decomposition form a factorization of is canonical. matrix math decomposition is in a Each problem. into of

mat

rix

dec

om

po

siti

on

mat

rix

fact

ori

zati

on

scie

nce

fic

tio

nca

no

nic

al f

orm



Visual expressions

“Inquisitor King” Inquisitor, King “King Inquisitor”

Giuseppe Verdi, Don Carlo, Metropolitan Opera


Geometric expressions


“Yellow Yellow”Yellow

No total order between points (only “far” and “near”)

Geometric expression = a pair of spatially close geometric words


Spatially-sensitive bags of features


is the probability

to find word at point and

word at point

Proximity between

points and

Distribution of pairs of geometric words

Shape distance

is the statistic of geometric expressions of the form



Spatially-sensitive bags of features


SHREC 2010 dataset


SHREC 2010 datasetBB et al, 3DOR 2010


ShapeGoogle with HKS descriptor (mAP %)BB et al, 3DOR 2010


ShapeGoogle with SI-HKS descriptor (mAP %)BB et al, 3DOR 2010


Scale 0.7 Heat Kernel Signature

?

Scale-Invariant Heat Kernel Signature

Scale-invariant retrieval

Kokkinos, B 2009


Scale 1.3 Heat Kernel Signature


Kokkinos, B 2009



Heat Kernel SignatureLocalscale


Kokkinos, B 2009



Structure

Local

Feature descriptors

Global

Metric

48Michael Bronstein Diffusion geometry for shape recognition

Beylkin & Niyogi 2003Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005Rustamov 2007

Laplacian embedding

Represent the shape using finite-dimensional Laplacian eigenmap

Ambiguities!


Osada, Funkhouser, Chazelle & Dobkin 2002Rustamov 2007

Global point signature (GPS) embedding

Deformation- and scale-invariant

No ambiguities related to eigenfunction permutations and sign

No need to compare multidimensional embeddings

Represent the shape using distribution of Euclidean distances in the

Laplacian embedding space (=commute time distances)

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45


Diffusion distance distributions

Mahmoudi & Sapiro 2009

Represent the shape using distribution of diffusion distances

Deformation-invariant How to select the scale?

0.5 1 1.5 2 2.5 3 3.5 x 10-3


Spectral shape distance

Kernel Distance Distribution Dissimilarity

Aggregation

BB 2010



Kernel Distance Distribution DissimilarityAggregation

BB 2010




Diffusion distance

BB 2010


Particular case I: Rustamov GPS embedding


BB 2010


Particular case II: Mahmoudi&Sapiro


BB 2010

Documents

1 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Feature-based methods and shape retrieval problems © Alexander