"Everything is related to everything else, but near things are more related to each other.”

-Waldo Tobler (1970)

First Law Of Geography:

Pyramids of FeaturesFor Categorization

Presented by Greg Griffin

Project Partner Will Coulter

Pyramids of FeaturesFor Categorization

Presented by Greg Griffin

Project Partner Will Coulter

Buckets of FeaturesFor Categorization

Presented by Greg Griffin

Project Partner Will Coulter

This talk is mostly about this paper:

With a little bit about benchmarking:

CALTECH 256

Images Features

€

rX 1K

r X Nx

€

rY 1K

r Y NY

€

K(X,Y )

A Number

Images Features

€

rX 1K

r X Nx

€

rY 1K

r Y NY

€

K(X,Y )

A Number

“How well dothey match”

“Weak Features”

“Weak Features”

(I think?)

“…points whose gradient magnitude in a given direction exceeds a minimum threshold.”

This is just their toy exampleThey use SIFT descriptors as “Strong Features”.

But you could use any features you want!

Images Features A Number

1 2 3 4 5 6 7 8

Images Features

X1 X5

Y1 Y5

Images Features

€

l D Il

Y1 Y5

X1 X5

Features

€

l D Il

€

K L (X,Y )

X1 X5

Y1 Y5

Features

€

l D Il

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

Features

€

l D Il

0 1

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

Features

€

l D Il

0 1 10

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

10

10

Features

€

l D Il

0 1 10

1 4

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

Features

€

l D Il

0 1 10

1 4

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

6

7

2

1

1

2

1

0

Features

€

l D Il

0 1 10

1 4 8

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

6

7

2

1

1

2

1

0

Features

€

l D Il

0 1 10

1 4 8

2 16

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

Features

€

l D Il

0 1 10

1 4 8

2 16 6

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

K L (X,Y )

Start By Matching Reds

5

2

1

1

23

1 1 1

1

€

L = 2

2

€

l D Il

0 1 10

1 4 8

2 16 6

€

K L (X,Y )

Start By Matching Reds

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ I l = 6 +8

21+

10

22

€

L = 2

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

l D Il

0 1 10

1 4 8

2 16 6

€

K L (X,Y )

Start By Matching Reds

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ I l = 6 +8

21+

10

22

€

L = 2

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

A compact set of matches is preferable to widely dispersed matches

€

l D Il

0 1 10

1 4 8

2 16 6

€

K L (X,Y )

Start By Matching Reds

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ I l = 6 +8

21+

10

22

€

L = 2

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

l D Il

0 1 10

1 4 8

2 16 6

€

K L (X,Y )

Start By Matching Reds

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ I l = 6 +8

21+

10

22

€

L = 2

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ ( I l − I l +1) = 6 +8 − 6

2+

10 − 8

22= 7.5

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

l D Il

0 1 10

1 4 5.5

2 16 1.9

€

K L (X,Y )

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ I l = 6 +8

21+

10

22

€

L = 2

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ ( I l − I l +1) = 5.4 < 7.5

A Sanity Check:Features X vs. Purely Isotropic Y

€

l D Il

0 1 10

1 4 8

2 16 6

€

K L (X,Y )

Start By Matching Reds, Then The Blues, Then…

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ I l = 6 +8

21+

10

22

€

L = 2

€

κ L (X1,Y1) = IL +1

2L−ll = 0

L−1

∑ ( I l − I l +1) = 6 +8 − 6

2+

10 − 8

22= 7.5

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

ΚL (X,Y ) = κ L (X m,Y m )m=1

M

∑ = 7.5 + ...

1 2 3 4 5 6 7 8

M = 8

Features

€

l D Il

0 1 10

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

Features

€

l D Il

0 1 10

1 4 8

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

Features

€

l D Il

0 1 10

1 4 8

2 16 6

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

κ L (X m,Y m ) = IL +1

2L−ll = 0

L−1

∑ ( I l − I l +1)

€

I l = min(HXl

i=1

D

∑ (i),HYl (i))

€

ΚL (X,Y ) =1

NxmNy

mκ L (X m,Y m )

m=1

M

∑

Foreach feature M=1…mForeach level = 0…L

Foreach cell i=1…D

€

l

Training Set

15Categories

100 Imagesper Category

Test Set

100-300 Imagesper Category

3.4

5.6

7.8

1.5

5.4

SL(X,Y)

office

store

coast

street

suburb

Confusion Matrix

Train on 100Test on 100-300

(per category)

Scene Database

Caltech 101

Hypothesis

Pyramid Matching works well when:

• Objects are aligned and localized– ie. certain Caltech 101 categories– biased by different values of Ntest?

• A few common features that define the category get randomly permuted through many positions, thanks to a large dataset– ie. scene database– now and then pyramid matching gets lucky

Test

How well will Pyramid matching work?

• Objects are not well aligned, or cluttered – Caltech 256 is more challenging in this respect

Scene Database

Caltech 101

Cluster Matching

k-means of SIFT positions

More Flexible Than A Grid?

k-means of SIFT positions + color

Position Invariance & Clutter

1. Any cluster can match any cluster2. Clusters respect duck / water boundaries

(sort of)

Tackles Alignment Problem

But… how to match efficiently?How many clusters? How big?

Summary

Spatial Pyramid Matching is efficient and handles a range of scales, but seems to be

sensitive to translation and clutter.

Cluster Matching has the potential to improve translational invariance and toleranceof clutter. But inefficient. Less principled: how

many clusters are optimal? How big should they be? No scale invariance.

Can we have the best of both worlds?

Try Sliding Spatial Pyramids?

Slide puzzle photo from:http://www.tenfootpolesoftware.com/products/slidepuzzle/