Part-based Object Retrieval with Binary Partition Trees

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)

X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 1

Part-based Object Retrievalwith Binary Partition Trees

Phd thesis dissertation by Xavier Giró-i-Nieto

Supervised by Professor Ferran Marqués Acosta (UPC)

and Professor Shih-Fu Chang (Columbia)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


AcknowledgementsJury GPI @ UPC

DVMM @ Columbia Students @ UPC

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction ←

2. Hierarchical Image Partitions

3. Interactive segmentation

4. Object Tree

5. Region Features

6. Codebooks

7. Fusion of Visual Modalities

8. Partition Tree Matching

9. Part-based Object Model

10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Problem statement

Content-Based Image Retrieval

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Problem statement

Semantic gap between user query and machine data.

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Challenges

Visual diversity within the same semantic class

anchor

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Challenges

Visual diversity within the same instance

Thisanchor

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Challenges

Semantics located at multiple scales

TV studioNews

Anchor

Jacket

Button

Head

Button

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Problem statement

This thesis focuses in the object retrieval with one example...

Queryobject

Targetdataset

Ranked list

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Problem statement

...and multiple examples.

Queryobjects

Target dataset

Ranked list

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction

2. Hierarchical Image Partitions ←


4. Object Tree

5. Region Features

6. Codebooks




10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


2. Hierchical Image Partition

Sliding windows

Image partition

Local scale

Interestpoints

Object analysis

ObjectSegmentation ? ✔✘✘

[Viola-Jones'01] SIFT [Lowe'04]

SURF [Bay'06]

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


✔Hierarchicalpartition


SpatialPyramidof points

Multi-scale analysis

Multiplesegmentations

Semantics at multiple scales

[Lazebnik'06][Bosch'07]

[Ravinovich'07]

[Vieux'10]

[Adamek'06]

[Gu'09]

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Adopted Solution: Binary Partition Trees (BPTs) [Garrido, Salembier 2000].


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Perceptual BPT

Semantic Object “head”

Split

Thesistopic


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction


3. Interactive segmentation ←

4. Object Tree

5. Region Features

6. Codebooks




10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


3. Interactive segmentationMotivations: Ground truth generation and retrieval interface.

Demo @ http://upseek.upc.edu/gat & http://upseek.upc.edu/gos

Graphical Annotation Tool (GAT) Graphical Object Searcher (GOS)

http://upseek.upc.edu/gat

http://upseek.upc.edu/gos

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree ←

5. Region Features

6. Codebooks




10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Related work: “Definition Hierarchy”, Jaimes and Chang (2001)

4. Object Tree

Thesisapproach

BPTnodes

BPTleaves

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Assumption: All objects are connected

Interactive segmentation BPT sub-roots selection

4. Object Tree

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


User selection of sub-roots on the

BPT.

Automatic top-down expansion through the

BPT

4. Object Tree

The Object Tree (OT) represents an instance of the object as a hierarchy of regions.

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features ←

6. Codebooks




10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


MPEG-7 and many more, “Visual Descriptors (VD)” (2001)

5. Region Features

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


5. Region Features

➔ Dominant Colors➔ Contour Shape➔ Edge Histogram

→ Region features in BPT

NOT Thesistopic

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree


6. Codebooks




10.Conclusions

Part I

Part II

Part III

Evaluation

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

How to evaluate an object retrieval system ?

1. Image Retrieval2. Object Detection3. Object Segmentation4. Search time

Annotated dataset

+ Metrics

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation (in Part II)

Annotated Dataset ETHZ: Category #1 “Apple logos”

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Annotated Dataset ETHZ: Category #2 “Bottles”


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Annotated Dataset ETHZ: Category #3 “Giraffes”


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Annotated Dataset ETHZ: Category #4 “Mugs”


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Annotated Dataset ETHZ: Category #5 “Swans”


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

1. Image Retrieval (global scale)

User Query (local) Ranked list of images (global)

Only global scale is considered.

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

P (k )=∣{relevant}∩{retrieved (k )}∣

k

R(k )=∣{relevant }∩{retrieved (k )}∣

∣relevant∣


Mean Average Precision (MAP) combines Precision and Recall for a set of queries.

Plotted by consideringdifferent k's

Precision

Recall

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


5. Region features

Image Retrieval (global scale)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

2. Object Detection (local scale-rough)

User Query (local) Retrieved boxes in relevant images

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

Pascal criterion requires a minimum of 50% overlapping in the union of detected object box and ground truth box.

2. Object Detection (local scale-rough)

Detection rate=∣{correct detections }∣

∣{relevant }∣

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Object Detection (local scale-rough)

5. Region features

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

3. Object Segmentation (local scale-precise)

User Query (local)Retrieved regions

from correct detections

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Evaluation

Jaccard index-J(A,B) compares retrieved region and ground truth mask at the pixel level.

J̄=1

∣D∣∑i∈D

J i

J (A , B)=∣A∩B∣∣A∪B∣

3. Object Segmentation (local scale-precise)

A B

Mean Jaccard Index combines the J's obtained by a set of correct detections D.

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Segmentation (local scale-precise)

5. Region features

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks ←




10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

Visual Word (VW): Vector quantization of observation (point, region...) onto a codebook [Sivic, Zissermann 2003].

Codebook

Visual Words

CodeRegion

Regions

(a1,a

2,a

3,a

4) (b

1,b

2,b

3,b

4) (c

1,c

2,c

3,c

4)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

Code Regions (CR) define the basis of the Parts Space.

“Regions described by their sub-parts”

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

Codebook

Visual Words

Regions

(1,0,0,0) ? (1,0,0,0) (0,1,0,0) ?

(a) Hard quantization

✘ ✘

SIFT [Sivic 2003]

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

Codebook

Visual Words

Regions

(0.25,0.25,0.25,0.25) (1,0,0,0)✘ ✘

(b) Probabilistic quantization

(0.25,0.25,0.25,0.25)

[Alpaydin 1998]

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

Codebook

Visual Words

Regions

(0,0,0,0) (1,0,0,0) ✘(0,0,0,0)

(c) Possibilistic quantization

✔

[Bezdek 1995]

Similarity to CRs

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

How to define the Parts Space with a given codebook ?

Possibilistic quantization alone characterizes regions, but not their sub-parts.

Region-basedPossibilistic

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

How to define the Parts Space with a given codebook ?

Bottom-up Max Pooling through BPT ✔ [Boureau'10]

FastExtraction

(max)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


BPT is exploited to describe sub-trees.

→ Hierarchy of Bags of Regions (HBoR)

6. Codebooks

FastExtraction

(max)

EfficientAnalysis(resilency to splits)

HBoR

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks

How to generate a codebook ? (1/2)

Train set

K-Means Clustering

Object parts

Codebook

BPT-based decomposition

CR1

CR2

CR3

CR4

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


a) Foreground and background regions - “generic”

b) Only object regions - “frgd”

6. Codebooks

How to generate a codebook ? (2/2)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


1. Image Retrieval (global scale) - Texture

6. Codebooks

HBoR features outstand region-based for shape & texture (not dominant colors).

Little differences between generic vs frgd codebooks (5 classes in ETHZ).

HBoR

Codebook size

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree


6. Codebooks

7. Fusion of Visual Modalities ←



10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Problem: Similarity scores between descriptors are not comparable.

Solution:

d color

x̄color

d shape d texture

x̄ shape x̄texture

x̄

Normalization (z-scores, SVM, w-scores)

Fusion (average, product, max/min)

5. Region Features + 7. Fusion

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks ←

7. Fusion of Visual Modalities ←



10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


6. Codebooks + 7. Fusion

Separate codebooks

When working with codebooks, when to apply fusion ?

Quantization

(a) Fusion after quantization - “nofused”

Cosinedistance

[colorshape

texture]

[ VW color

VW shape

VW texture]

Fusion[ x̄colorx̄ shapex̄ texture

] x

Queryregion

Targetregion

[ VW color

VW shape

VW texture]

[colorshape

texture]

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Fused codebook


(b) Fusion during quantization - “fused”


Cosinedistance

[colorshape

texture]

[VW ]

x

Queryregion

Targetregion

[VW ]

[colorshape

texture]Quantization

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)




w-scoresz-scores


W-scores outperform z-scores when working with HBoR.

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks


8. Partition Tree Matching ←

9. Part-based Object Model ←

10.Conclusions

Part I

Part II

Part III Evaluation

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


CCMA News dataset.

Topology diversity on 11 anchors objects.

Evaluation (in Part III)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks


8. Partition Tree Matching ←


10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Reminder: The Object Retrieval Problem (single instance)

Queryobject

Targetdataset

Ranked list


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Object Tree (OT) Target BPT

?Q2Q1

Q0

Q4Q3

Challenges:

(a) Distance for a given OT-BPT correspondence

(b) Matching algorithm


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Proposal (ad-hoc): Weighted sum of part matches

8. Partition Tree Matching: Distance

αi=1

max (1,∣Ci∣)

area(i)

area(Pi)

Example:Children of part i

Parents of part i

Q2Q1

Q0

d=∑i=0

N

wi di wi=i ∏j∈Ancestors Qi

j=i⋅wParent Qi, where

α0=1 /2=w0α1=area(Q1)/area (Q0)α2=area(Q2)/area(Q0)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Two proposals:

8. Partition Tree Matching: Algorithm

Object Tree (OT) Target BPT

?Q2Q1

Q0

Q4Q3

(a) Top-down Matching

(b) Bottom-Up Matching

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Assumption: All OT nodes are contained in a single sub-BPT.

Assumption: OT nodes are spread through different sub-BPTs.

- common

+ common(b) Bottom-Up Matching



Sets a baseline

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Q2Q1

Q0

Assumption: The best match for the OT is contained in a single sub-tree of the target BPT.

Target BPTOT

Q2Q1

Q0


(b) Top-down QT Matching

1. Set a match for Q0

2. Find matches for Q1 and Q2

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


What is the impact of decomposing the query into parts ?


Retrieval gain until the “correct” amount of parts (3).

Image Retrieval (global scale)

MPEG-7Region Top-down

match

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


What is the impact of decomposing the query into parts ?


Exponential growth of search time with the amount of parts.

Search time

Top-downmatch

MPEG-7Region

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Assumption: All OT nodes are contained in a single sub-BPT.

Assumption: OT nodes are spread through different sub-BPTs.

- common

+ common(b) Bottom-Up Matching



Sets a baseline

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Q2Q1

Q0

Weaker assumption that in the top-down case:

This regionis not part

of the target BPT

→ OT leaves are among the nodes of the target BPT

(c) Bottom-UP QT Matching


Q2Q1

Q0

1. Set a match for Q1 & Q2

2. Assess the union o f candidates for Q1 and Q2 to match Q0

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


How to rapidly describe the merged region generated during search time ?

(c) Bottom-UP QT Matching


(1, 1, 1, 1)

→ Approximate HBoR satisfies this requirement.

Q2Q1

Q0

Object Tree

HBoR

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


What is the impact of introducing the bottom-up match ?


A clear gain, the object splits in the target BPTs are solved.

RegionVW

HBoR

Bottom-upgain

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks



9. Part-based Object Model ←

10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Queryobjects

Target dataset

Ranked list


Reminder: The Object Retrieval Problem (multiple instances)

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Query objects Model

Training

Target image

Test

Retrieved object

Model-based retrieval


Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts

Inclusion classifier

Detector classifier

Fusion classifier

# Parts

ContextualfilteringT

est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts


Detector classifier

Fusion classifier

# Parts


est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Unsupervised clustering with Quality Threshold [Heyer 1999].

Two parameters: - Mininum amount of elements in cluster- Maximum radius of a cluster

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts


Detector classifier

Fusion classifier

# Parts


est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Positive: Parts and their ancestorsNegative: Parts' children and positive node's siblings.


Inclusion classifier (training)

HBoR

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts


Detector classifier

Fusion classifier

# Parts


est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Efficient top-down exploration of the target BPT.


Inclusion classifier (test)

Not explored(efficiency)

HBoR

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts


Detector classifier

Fusion classifier

# Parts


est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Positive: Parts.Negative: Random sampling (balanced set).


Detector classifier (training)

Region-basedPossibilistic

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts


Detector classifier

Fusion classifier

# Parts


est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Positive training samples represent valid combinations of parts.Negative training samples represent invalid combinations.

Parts of object i Parts of object j

Type 1 Type 2 Type 3 Type 4

(1, 1, 0, 0) (0, 1, 1, 1)


Fusion classifier (training)

Parts definition

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



Parts definition

Tra

inin

g

# Parts


Detector classifier

Fusion classifier

# Parts


est

Fusion classifier


Detector classifier

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



How important is a correct estimation of the types of parts ?

The performance is dependent from the Quality Threshold parameters.

CodeBooksize

Image

Retrieval

Min # elems in cluster

Max radius for cluster

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)



What is the impact of codebook size in terms of time ?

Search time does not grow with codebook size (Efficient exploration ??).

Training

time

Search

time

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks




10.Conclusions ←

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Conclusions

BPT presents several opportunities for local analysis:

Three main contributions in this thesis:

→ Interactive segmentation→ Semantic analysis

Contribution#1

Contribution#2

Contribution#3

Hierarchy of Bags of Regions:● Possibilistic quantization● Max pooling throug BPT

Object Tree - BPT matching● Top-down● Bottom-up

Part-based model for objects● Automatic determination of parts● Efficient BPT analysis.

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Open work

Test in public benchmark.

→ Instance Search task @ TRECVID 2012

Richer image representations

→ Amount of BPT leaves adapted to content→ Discard uninformative merges & allow multiple options

Enrich region descriptors

Better model for combination of parts representing object

→ Introduce interest points→ Significance estimation

Local segmentation of objects from global annotations

Smarter codebooks

→ Unsupervised clustering→ TF-IDF weights & Stop words

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Outline

1. Introduction



4. Object Tree

5. Region Features

6. Codebooks




10.Conclusions

Part I

Part II

Part III

Gru

p d

e P

rocess

am

en

t d

e la Im

atg

eU

niv

ers

itat

Po

litè

cn

ica d

e C

ata

lun

ya (

UP

C)


Thank you, moltes gràcies !

Technology

Part-based Object Retrieval with Binary Partition Trees