View
230
Download
0
Category
Tags:
Preview:
Citation preview
Outline
• Overview of image retrieval/search– Basic paradigm – Local features indexed by vocabulary trees– Global features indexed by compact hash codes
• Query specific fusion– Graph construction– Graph fusion– Graph-based ranking
• Experiments
Content-based Image retrieval/search
Features Database
Images Hashing codes
feature extraction hashing indexing
Inverted indexing
FeaturesQuery image
Hashing codes
feature extraction hashing search
Rank list
re-rank
Offline
Online
• Scalability !!!– Computational efficiency– Memory consumption– Retrieval accuracy
Local Features Indexed by Vocabulary Trees
• Features: SIFT features• Hashing: visual word IDs by
hierarchical K-means.• Indexing: vocabulary trees• Search: voting and sorting
Scalable Recognition with a Vocabulary TreeD. Nister and H. Stewenius, CVPR’06
• An example:• ~1K SIFT features per image• 10^6~1M leaf nodes in the tree• Query time: ~100-200ms for 1M
images in the database
Global Features Indexed by Compact Hash Codes
• Feature: GIST, RGB or HSV histograms, etc.• Hashing: compact binary codes, e.g., PCA+rotation+binarization.• Indexing: a flat storage with/out inverted indexes• Search: exhaustive search with Hamming distances + re-ranking
Modeling the Shape of the Scene: A Holistic Representation, A. Oliva and A. Torralba, IJCV’01Small Codes and Large Image Databases for Recognition, A. Torralba, R. Fergus, Y. Weiss, CVPR’08Iterative Quantization: A Procrustean Approach to Learning Binary Codes, Y. Gong and S. Lazebnik, CVPR’11
• An example:• GIST -> PCA -> Binarization• 960 floats -> 256 floats -> 256
bits (217 times smaller).• Query time: 50-100ms, search
1M images using Hamming dist
Motivation• Pros and Cons
• Can we combine or fuse these two approaches?• Improve the retrieval precision• No sacrifice of the efficiency• Early fusion (feature level)?• Late fusion (rank list level)?
Retrieval speed
Memory usage
Retrieval precision
applications Image properties to attend to
Local feat. fast high high near duplicate local patterns
Global feat. faster low low general images global statistics
Challenges• The features and algorithms are dramatically different.– Hard for the feature-level fusion– Hard for the rank aggregation
• The fusion is query specific and database dependent– Hard to learn how to combine cross different datasets
• No supervision and relevance feedback!– Hard to evaluate the retrieval quality online
Query Specific Fusion• How to evaluate online the quality of retrieve results
from methods using local or global features?• Assumption: The consensus degree among top
candidate images reveal the retrieval quality– The consistency of top candidates’ nearest neighborhoods.
• A graph-based approach to fusing and re-ranking retrieval results of different methods.
Graph Construction• Construct a weighted undirected graph to represent
a set of retrieval results of a query image q.• Given the query q, image database D, a similarity
function S(.,.), top-k neighborhood .• Edge: the reciprocal neighbor relation
• Edge weight: the Jaccard similarity between neighborhoods
Graph-based Ranking• Ranking by a local Page Rank– Perform a link analysis on G– Rank the nodes by their connectivity in G
• Ranking by maximizing weighted density
Experiments• Datasets: 4 public benchmark datasets– UKBench : 2,550*4=10200 images (k=5)– Corel-5K : 50*100 = 5000 images (k=15)– Holidays : 1491 images in 500 groups (k=5)– SFLandmark : 1.06M PCI and 638K PFI images (k=30)
• Baseline methods– Local features: VOC (contextual weighting, ICCV11)– Global features: GIST (960D=>256bits), HSV (2000D=>256bits)– Rank aggregation– A fusion method based on an SVM classifier
• Nearest neighbors are stored offline for the database
UKBench• Evaluation: 4 x recall
at the first four returned images, referred as N-S score (maximum = 4).
Corel-5K• Corel 5K: 50 categories, each category has 100 images.
Average top-1 precision for leave-one-out retrievals.
San Francisco Landmark• Database images: – Perspective central images (PCIs): 1.07M – Perspective frontal images (PFIs): 638K.
• Query images: 803 image taken with a smart phone• Evaluation: The recall rate in terms of buildings
Computation and Memory Cost
• The average query time
• Memory cost– 340MB extra storage for the top-50 nearest
neighbor for 1.7M images in the SFLandmark.
Conclusions
• A graph-based query specific fusion of retrieval sets based on local and global features– Requires no supervision– Retains the efficiency of both methods– Improves the retrieval precision consistently on 4 datasets– Easy to be reproduced by other motivated researchers
• Limitations– No reciprocal neighbor for certain queries in either
methods– Dynamical insertion or removal of database images
Recommended