Glasgow 02/02/04

NNk networks for content-based image retrieval

Daniel Heesch

Overview

• CBIR – a problem of image understanding?

• Approaches to feature weighting

• The NNk technique for retrieval

• Getting connected: NNk networks for browsing

• Future work

Challenges: Semantic gap

What is the relationship between low-level features and image meaning?

Challenges: Image polysemy

one image - multiple meanings

Feature Weights

i

ii YXswYXS ),(),(

Approaches to feature weighting (1)

Post retrieval relevance feedback• effectiveness relies on a good first retrieval result• useful for fine-tuning weights

Approaches to feature weighting (2)

SVM Metaclassifier (Yavlinsky et al., ICASSP 2004)

• Given a set of queries and ground truth, for each query:• sample at random m positive and negative images and

for each build a score vector consisting of the feature-specific similarities between that image and the query

• Use an SVM to obtain a hyperplane that separates positive and negative score vectors with least error

• For an unseen image, compute similarity to the query as the distance of its score vector to the hyperplane.

Approaches to feature weighting (2)• The distance of a vector to a hyperplane is a weighted sum

of the vector’s components, which is just the aggregation formula shown to you previously.

• The hyperplane represents a set of feature weights that maximise the expected mean average precision

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Colour

Te

xtu

re Positive

Negative

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Colour

Co

nv

olu

tio

n F

ilte

rs

Positive

Negative

Approaches to feature weighting (2)• ~ 300 score vectors needed to establish near-optimal weights for a subset of the

Corel collection on average (6192 images)• No query-specific weight optimization

Combination Performance

0.34

0.35

0.36

0.37

0.38

0.39

0.4

2 3 4 5 6 7

Number of n -best features fused

M.A

.P Linear SVM

CombSum

Approaches to feature weighting (3)

Query-specific optimization (Aggarwal et al., IEEE Trans. Multimedia 2002)

• Modify query representation along each feature axis and regenerate modified query, ask user whether new query image is still relevant

• Interesting idea but limited applicability in practice

NNk retrieval

The idea of NNk – a two-step approach

1. Retrieve with all possible weight sets -> returns a set of images (NNk) each associated with a particular weight

2. Retrieve with the weights associated with the relevant images the user selects

The first retrieval step: finding the NNk

• For each feature combination w, determine nearest neighbour of the query

• Record for each nearest neighbour the proportion of w for which it came top as well as the average of these w

• NN for nearest neighbour, k for the dimensionality of the weight space(= length of the weight vector w)

F2 F1

F3

The first retrieval step: finding the NNk

1

11

The first retrieval step: finding the NNk

The first retrieval step: finding the NNk

• With fixed number of grid points, time complexity is exponential in the number of features (k)

• Useful theorem:

if for any two weight sets w1 and w2 that differ only in two components the top ranked image is the same, then this image will be top ranked for all linear combinations of w1

and w2

The first retrieval step: finding the NNk

Visualization of NNk

• Retrieve with each weight set in turn

• Merge ranked lists

The second retrieval step:

• Comparison of NNk with two post-retrieval methods for weight-update1. Our own: minimize

2. Rui’s method (Rui et al., 2002)

Performance evaluation

Performance evaluation

• Corel Gallery 380,000 Package

• Given a subset of images, treat each image as a query in turn and retrieve from the rest

• For RF: retrieve with equal weight sets, gather relevance data and retrieve with new weight set

• For NNk: determine NNk, gather relevance data and retrieve with new weight sets

• Determine MAP after second retrieval

Performance evaluation

NNk networks

Network Construction

• Vertices represent images• An arc is established between two images X

and Y, iff there exist at least one instantiation of the weight vector w, for which Y is the nearest neighbour of X

• Record for each nearest neighbour the proportion of w, for which it came top-> edge weight, measure of similarity

• Storage: for each image, its nearest neighbours and their frequencies

Rationale

• exposure of semantic richness• user decides which image meaning is the

correct one• network precomputed -> interactive• supports search without query formulation

Graph topology: small world properties

• small average distance between any two vertices (three nodes for 32000 images)

• high clustering coefficient: an image‘s neighbours are likely to be neighbours themselves

Corel Sketches Video

k 5 4 7

Vertices 6,192 238 32,318

Edges 150,776 1,822 1,253,076

C(G) 0.047 0.134 0.14

Crand(G) 0.004 0.03 0.0012

Dist 3.22 3.29 3.33

Distrand 2.73 2.68 2.83

Graph topology: scale-freeness

• Degree distribution follows power-law

Image Browsing

• Initial display: retrieval result using search-by-example

OR cluster display using Markov-Chain Clustering

(MCL) technique (van Dongen, 2000)• Clicking on an image displays all adjacent

vertices in the network• Distance inversely proportional to edge

weight

Evaluation of NNk networks: TRECVID2003

• search collection: 32000 keyframes from news videos

• 24 topics: example images + text

• Four system variants: Search + Relevance Feedback + BrowsingSearch + Relevance FeedbackSearch + BrowsingBrowsing

Future work

User interaction

• What can we infer from the history of images in the user‘s search path?The location of the target? Changing information needs?

Network construction and analysis

• Hierarchical sampling of points in weight space

• Incremental update of network while preserving small-world properties

• Optimal network structures

Thanks