Content Based Image Retrieval Techniques

Ambrose Tuscano (atuscan1@umbc.edu)University of Maryland Baltimore County,

CMSC 676 Information Retrieval

Introduction

Image retrieval systems aim to find similar images to a query image among an image dataset.

Represented as

• Pixels (Also called Rasters)

• Vectors

● By annotation (manual) • Text retrieval • Semantic level (good for picture with people, architectures)

● By the content (automatic) • Color, texture, shape • Vague description of picture (good for pictures of scenery and with pattern and texture)

Features in an Image

● Color : Low level, Can't specify context.

● Texture : Produce a mathematical characterisation of a repeating pattern in the image.

● Shape: Region based and Contour(outline) based.

● Local Image Features : small parts of a big image. ○ extracted from the images at salient points and dimensionality

reduced using Principal Component Analysis (PCA) transformation○ SIFT using Harris interest points

Structure

IMAGE RETRIEVAL METHODS

Text based Image RetrievalFirst annotated the images by text and then used text-based database management systems to perform image retrieval.

Text based Image Retrieval

Three Ways to go● Manually Assign Keywords to each image

● Use text associated with the images (captions, web pages)

● Analyse the image content to automatically assign keywords(Computer Vision?)

Content Based Image Recognition

● A technique which uses visual contents to search images from large scale image databases according to users' interests.

● CBIR research is mainly contributed by the computer vision community

Content based Image Recognition

To use local features for image retrieval, three different methods are available:

● Direct transfer: nearest neighbors for each of the local features of the query searched and the database images containing most of these neighbors returned.

● Local feature image distortion model (LFIDM): Compares the distances between local features from the query image to the local features of each image of the database . The images with the lowest total distances are returned.

● Histograms of local features: A reasonably large amount of local features from the database is clustered and then each database image represented by a histogram of indices of these clusters.

● Color Histogram● Color Correlogram● Color AutoCorrelogram● Color Coherence vector● Dominant Color Descriptors

● A shape is the form of an object or its external boundary, outline,or external surface, as opposed to other properties like color, texture or composition.

● Fourier Descriptors● Canny Algorithm● SIFT Descriptors● Moment Invariants● Eccentric and Axis Oriented

●

●●●●

●●●

○

●○○○

● Smoothing: Blur image to remove Noise

● FInd Gradients : Edges are marked where gradients of image have large magnitudes.

● Non-Max Suppression: Only local Maxima is marked for edges.

● Double Thresholding: Potential Edges are determined

● Hysteresis : Finally Edges which are not connected/near to many other potential edges are removed.

Texture Extraction- Motif Co-Occurrence Matric● MCM is used to represent transveral

of adjacent pixel color difference in an image.

● Each Pixel corresponds to four adjacent pixel colors

● Each image can be presented by four images of motifs of scan pattern, which can be further constructed into four two dimensional matrices of the image size.

● The attribute of the image will be computed with motifs of scan pattern and a color motif cooccurence matrix(CMCM) will be obtained

● Euclidean DIstance● Mahalanobis Distance● MInkowski Distance● Histogram Intersection Distance● Quadratic Form Distance

Techniques used by CBIR

● K-MeansK-means clustering algorithm is proposed as it improves the scalability.

● Wavelet TransformFeature vectors of images are be constructed from wavelet transformations,

which can also be utilized to distinguish images through measuring distances between feature vectors.

● Support Vector Machine: SVM classifier can be trained using training data of images marked by users .

● Neural Network:A CNN doesn’t need complex work like feature extraction to work. Having a

proper labelled data, we can train the system to learn the data features using complex layer structure.

CBIR + TBIR◦CBIR can be costly in the fact that it needs a lot of complex computations.

◦TBIR can be comparatively fast but has low precision.◦A hybrid model is currently being implemented.

◦a text-based image meta-search engine retrieves images from the Web using the text information from the Query.

◦ Techniques like matching term frequency-inverse document frequency (tf-idf) weightings and cosine similarity are used.

◦use the CBIR approach to re-filter the search results.

● X.Y. Wang,Y.J. Hong and H.Y.Yang,”An effective image retrieval scheme using color,

texture and shape features”

● Nidhi Singh ,Kanchan Singh and Ashok Sinha “A Novel Approach for Content Based Image

Retrieval”

● Yogita Mistry,and D. T. Ingole “Survey on Content Based Image Retrieval Systems”

● John Canny, “A Computational Approach to Edge Detection”

● Mussarat Yasmin, Muhammad Sharif and Sajjad Mohsin, “Use of Low Level Features for

Content Based Image Retrieval: Survey”

● N. Jhanwar, Subhasis Chaudhuri, et.al. Content based image retrieval using motif

cooccurrence matrix