Upload
jonathon-hare
View
127
Download
0
Tags:
Embed Size (px)
Citation preview
Content-based image retrieval using a mobile
device as a novel interface
Jonathon S. Hare and Paul H. Lewis
Intelligence, Agents, Multimedia GroupDepartment of Electronics and Computer Science
University of Southampton{jsh02r, phl}@ecs.soton.ac.uk
Introduction
• This work investigates the use of a mobile device as a novel interface to a image retrieval system.
• We develop a new two-stage retrieval strategy that exhibits impressive retrieval performance even with the poor imaging quality of the camera on the mobile device.
Content-based Image Retrieval
• A retrieval system that is robust to the poor imaging qualities found in low-end digital cameras.
• Two stage algorithm:
• First stage inspired by Information Retrieval techniques: Vector-Space model.
• Second stage re-ranking of the first stage results based on a geometric constraint.
Salient Regions for CBIR
• Retrieval based on DoG based salient regions.
• DoG SR’s previous shown to be robust to noise, rotation and other degradation's, such as those found in low-end digital cameras.
Local Descriptors
• Lowe’s SIFT Key feature used to describe image in each salient region.
• Robust to small shifts in salient region position.
• Robust to illumination changes.
• Doesn’t use colour.
Retrieval Using Text Retrieval Techniques
• We have tried to apply techniques taken from the information retrieval field:
• Vector-Space Model.
• Documents in the collection are represented by vectors of term-frequency.
Retrieval Using Text Retrieval Techniques
• A vocabulary of ‘visual words’ is built, based on feature vectors from a subset of the images in the database.
• Each feature vector in the database is vector quantised to the closest ‘visual word’.
Retrieval Using Text Retrieval Techniques
• Vector-Space Model.
• Term frequency vectors are weighted:
• TF-IDF.
• Weighted Vector is created for the query image. Documents are ranked by comparing vectors using cosine similarity.
Geometric Consistency
• Second stage of retrieval involves re-ranking based on geometric constancy of matching salient regions.
• Use RANSAC to estimate planar homography between query and target image, then re-rank on percentage of salient region pairs fitting the homography.
Mobile Client-Server Implantation
XML-RPC Web Service
Query Engine
Salient Region & Feature Vector
Generator
Feature Vector and Metadata Database
XML-RPC Encoded query, with embedded
compressed image
XML-RPC Encoded response, with an embedded URL
Retrieval Performance
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0 20 40 60 80 100
Pe
rce
nta
ge
Rank
CCVColour HistogramMono Histogram
PWTVectorSpace
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90 100
Pe
rce
nta
ge
N
% Correctly matched% Incorrectly matched
Performance of retrieval with second-stage, versus number of first-stage results considered
Performance of first-stage retrieval, versus rank of matching image
Future Work• Investigate how to construct an optimal
‘visual’ vocabulary.
• Add a local colour-based descriptor to use in addition to the SIFT descriptors.
• Investigate the use of “stop-words” and their effect on retrieval performance.
• Implement an inverted index to aid efficiency and speed.
Conclusions• We have developed a two-stage image
retrieval algorithm that is able to effectively retrieve correctly matching images with query images from low-quality sources, such as cheap digital cameras.
• The system has been implemented in a client-server fashion, with a mobile device used for generating queries and receiving results.