6/9/2010 - Techniontosca.cs.technion.ac.il/book/handouts/TechnionEE2010_feature.pdf · 6/9/2010 9 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems

6/9/2010

1

1Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems

Feature-based methodsand shape retrieval problems

© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book

048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes

EE Technion, Spring 2010


Structure

Local

Feature descriptors

Global

Metric


Combining local and global structures

BBK 2008; Keriven, Torstensen 2009; Dubrovina, Kimme l 2010; Wang, B 2010

Pair-wise stress (global) Point-wise stress (local)

Local structure can be geometric or photometric


Photometric stress

Thorstenstein & Keriven 2009

5Numerical Geometry of Non-Rigid Shapes Diffusion Geometry

Heat kernels, encore

Brownian motion on X starting at point x, measurable set C

probability of the Brownian motion to be in C at time t

Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005

Heat kernel represents transition probability


Intrinsic descriptors

Sun, Ovsjanikov & Guibas 2009

Multiscale local shape descriptor (Heat kernel signa ture)

can be interpreted as probability of Brownian motion to return to

the same point after time (represents “stability” of the point)

Time (scale)

6/9/2010

2


Sun, Ovsjanikov & Guibas 2009for small t

Relation to curvature


Heat kernel signature

Heat kernel signatures represented in RGB space

Sun, Ovsjanikov & Guibas 2009Ovsjanikov, BB & Guibas 2009


Heat kernel descriptors

Invariant to isometric deformations Localized sensitivity

to topological noise

J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009M. Ovsjanikov, BB, L. Guibas, 2009


Scale invariance

Original shape Scaled by

HKS= HKS=

Not scale invariant!


Scale-invariant heat kernel signature

B, Kokkinos CVPR 2010

Log scale-space

Scaling = shift and multiplicative

constant in HKS

log + d/d ττττ

Undo scaling

Fourier transformmagnitude

Undo shift

0 100 200 300-15

-10

-5

0

τ0 100 200 300

-0.04

-0.03

-0.02

-0.01

0

τ0 2 4 6 8 10 12 14 16 18 20

0

1

2

3

4

ω=2kπ/T


Scale invariance

B, Kokkinos 2009

Heat Kernel Signature Scale-invariantHeat Kernel Signature

6/9/2010

3


Bending invariance

B, Kokkinos CVPR 2010

Heat Kernel Signature Scale-invariantHeat Kernel Signature


Bending invariance

Wang, B 2010

Geodesic+HKS Diffusion+HKS Commute+SI-HKS


Topology invariance

Geodesic+HKS Diffusion+HKS

Wang, B 2010


Scale invariance

Wang, B 2010

Geodesic+HKS Commute+SI-HKS


Invariance

Geodesic metric

Rigid Inelastic Topology

Diffusion metric

Scale

Wang, B 2010

Commute-timemetric

Heat kernelsignature (HKS)

Scale-invariant HKS (SI-HKS)


6/9/2010

4


Tagged shapes

Shapes withoutmetadata

Man, person, humanPersonText search

Content-based search

3D warehouse


?

Content-based search problems

Invariant shape retrievalShape classification

?

Semantic

Variability of shape

within category

Geometric

Variability of shape

under transformation


Image vs shape retrieval

Illumination View Missing data

Deformation Topology Missing data


Bags of words

Notre Dame de Paris is a Gothic cathedral in the fourthquarter of Paris, France. It was the first Gothicarchitecture cathedral, and its construction spannedthe Gothic period.

cons

truc

tion

arch

itect

ure

Italy

Fra

nce

cath

edra

lch

urch

basi

lica

Par

isR

ome

Got

hic

Rom

an

St. Peter’s basilica is the largest church in world,located in Rome, Italy. As a work of architecture, it isregarded as the best building of its age in Italy.

Notre Dame de Paris is a Gothic cathedral in the fourthquarter of Paris, France. It was the first Gothicarchitecture cathedral, and its construction spannedthe Gothic period.

St. Peter’s basilica is the largest church in world,located in Rome, Italy. As a work of architecture, it isregarded as the best building of its age in Italy.


Bags of features

Visual vocabulary

Feature detector + descriptor

Invariant to changes of the image

Discriminative (tells different images apart)


Advantages

� “Shape signature”

� Easy to store

� Easy to compare

� Partial similarity possible

6/9/2010

5


Images vs shapes

Images Shapes

Many prominent features Few prominent features

Affine transforms, illumination,

occlusions, resolution

Non-rigid deformations, topology,

missing parts, triangulation

SIFT, SURF, MSER, DAISY, … Curvature, conformal factor,

local distance histograms


“ShapeGoogle”

Feature descriptor

Geometric words

Bag of words

Geometric expressions

Spatially-sensitive bag of features

“ ”

“ ”


Geometric vocabulary

M. Ovsjanikov, BB, L. Guibas, 2009


Bags of features



Nearest neighbor in the descriptor space


Bags of features



Weighted distance to words

in the vocabulary


Bags of features

Shape distance = distance between bags of features


Statistics of different geometric words over the entire shape

6/9/2010

6


Index in vocabulary1 64


Bags of features

32Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Statistical weighting

Query q Database D

syzygy in astronomy means alignment ofthree bodies of the solar system along astraight or nearly straight line. a planet isin syzygy with the earth and sun when itis in opposition or conjunction. the moonis in syzygy with the earth and sun whenit is new or full.

syzygy in astronomy means alignment ofthree bodies of the solar system along astraight or nearly straight line. a planet isin syzygy with the earth and sun when itis in opposition or conjunction. the moonis in syzygy with the earth and sun whenit is new or full.

Sivic & Zisserman 2003BB, Carmon & Kimmel 2009

Frequent in document= important

in is

or

syzygy

Rare in database= discriminative

with

a

of

the

and when

33Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Statistical weighting

Query q Database D

Significance of a term t

Term frequency Inverse documentfrequency

Weight bags of features by tf-idf

Reduce the influence of non-important terms in dense descriptor

Sivic & Zisserman 2003BB, Carmon & Kimmel 2009


Expressions

In math science, matrixdecomposition is afactorization of a matrixinto some canonicalform. Each type ofdecomposition is used ina particular problem.

In biological science,decomposition is theprocess of organisms tobreak down into simplerform of matter. Usually,decomposition occursafter death.

Matrix is a science fictionmovie released in 1999.Matrix refers to asimulated reality createdby machines in order tosubdue the humanpopulation.

mat

rix d

ecom

posi

tion

mat

rix fa

ctor

izat

ion

scie

nce

fictio

nca

noni

cal f

orm


In biological science,decomposition is theprocess of organisms tobreak down into simplerform of matter. Usually,decomposition occursafter death.


mat

rixde

com

posi

tion is a

the of in to by

scie

nce

form





Expressions


mat

rixde

com

posi

tion is a

the of in to by

scie

nce

form

In particular matrix usedtype a some science,decomposition form afactorization of iscanonical. matrix mathdecomposition is in aEach problem. into of

mat

rix d

ecom

posi

tion

mat

rix fa

ctor

izat

ion

scie

nce

fictio

nca

noni

cal f

orm



Visual expressions

“Inquisitor King” Inquisitor, King “King Inquisitor”

Giuseppe Verdi, Don Carlo, Metropolitan Opera

6/9/2010

7


Geometric expressions


“Yellow Yellow”Yellow

No total order between points (only “far” and “near”)

Geometric expression = a pair of spatially close geometric words


Spatially-sensitive bags of features


is the probability

to find word at point and

word at point

Proximity between

points and

Distribution of pairs of geometric words

Shape distance

is the statistic of geometric expressions of the form



Spatially-sensitive bags of features


SHREC 2010 dataset


SHREC 2010 datasetBB et al, 3DOR 2010


ShapeGoogle with HKS descriptor (mAP %)BB et al, 3DOR 2010

6/9/2010

8


ShapeGoogle with SI-HKS descriptor (mAP %)BB et al, 3DOR 2010


Scale 0.7 Heat Kernel Signature

?

Scale-Invariant Heat Kernel Signature

Scale-invariant retrieval

Kokkinos, B 2009


Scale 1.3 Heat Kernel Signature


Kokkinos, B 2009



Heat Kernel SignatureLocalscale


Kokkinos, B 2009



Structure

Local

Feature descriptors

Global

Metric

48Michael Bronstein Diffusion geometry for shape recognition

Beylkin & Niyogi 2003Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005Rustamov 2007

Laplacian embedding

Represent the shape using finite-dimensional Laplacian eigenmap

Ambiguities!

6/9/2010

9


Osada, Funkhouser, Chazelle & Dobkin 2002Rustamov 2007

Global point signature (GPS) embedding

☺ Deformation- and scale-invariant

☺ No ambiguities related to eigenfunction permutations and sign

☺ No need to compare multidimensional embeddings

Represent the shape using distribution of Euclidean distances in the

Laplacian embedding space (=commute time distances)

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45


Diffusion distance distributions

Mahmoudi & Sapiro 2009

Represent the shape using distribution of diffusion distances

☺ Deformation-invariant � How to select the scale?

0.5 1 1.5 2 2.5 3 3.5 x 10-3


Spectral shape distance

Kernel Distance Distribution Dissimilarity

Aggregation

BB 2010



Kernel Distance Distribution DissimilarityAggregation

BB 2010




Diffusion distance

BB 2010


Particular case I: Rustamov GPS embedding


BB 2010

6/9/2010

10


Particular case II: Mahmoudi&Sapiro


BB 2010

Documents

6/9/2010 - Techniontosca.cs.technion.ac.il/book/handouts/TechnionEE2010_feature.pdf · 6/9/2010 9 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems