1

Content-Based Image Retrieval

2

What is Content-based Image Retrieval (CBIR)?

• Image Search Systems that search for images by image content<-> Keyword-based Image/Video Retrieval

(ex. Google Image Search, YouTube)

3

How does CBIR work ?

• Extract Features from Images• Let the user do Query

– Query by Sketch– Query by Keywords– Query by Example

• Refine the result by Relevance Feedback– Give feedback to the previous result

4

Query by Example

Query Sample

Results

CBIRCBIR

“Get similar images”

• Pick example images, then ask the system to retrieve “similar” images.

What does “similar” mean?

5

Relevance Feedback

• User gives a feedback to the query results• System recalculates feature weights

Initialsample 1st Result

Query

2nd Result

Feedback Feedback

6

Two Classes of CBIRNarrow vs. Broad Domain

• Narrow– Medical Imagery Retrieval– Finger Print Retrieval– Satellite Imagery Retrieval

• Broad– Photo Collections– Internet

7

The Architecture of a typical CBIR System

ImageManager

ImageManager Image

Database

Image Database

Multi-dimensional Indexing

Feature Extraction

Feature Database

Feature Database

Retrieval Module

Retrieval Module

User InterfaceUser Interface User InterfaceUser Interface

8

Feature Extraction for the Query Image

The Retrieval Process of a typical CBIR System

Image Database

Image Database

Feature Database

Feature Database

Feature Extraction 1

Feature Extraction 2

Feature Extraction n

(2 3 5 0 7 9)Feature Vector 1

(1 1 0 0 1 0)Feature Vector 2

(33.5 6.7 28.6 11.8 5.5)Feature Vector n Sorting Image

Manager

ImageManager

SimilarityComparison

(Image ID, similarity) Images

Query Image

InterfaceManager

InterfaceManager

Results

9

Basic Components of CBIR

– Feature Extraction– Data indexing– Query and feedback processing

10

How Images are represented

11

Image Features

• Representing the Images– Segmentation – Low Level Features

• Color• Texture• Shape

12

Image Features

• Information about color or texture or shape which are extracted from an image are known as image features– Also a low-level features

• Red, sandy

– As opposed to high level features or concepts• Beaches, mountains, happy

13

Global features

• Averages across whole image Tends to loose distinction between foreground

and background Poorly reflects human understanding of images Computationally simple A number of successful systems have been

built using global image features

14

Local Features

• Segment images into parts• Two sorts:

– Tile Based– Region based

15

Regioning and Tiling Schemes

(a) 5 tiles (b) 9 tiles

(c) 5 regions (d) 9 regions

Tiles

Regions

16

Tiling

Break image down into simple geometric shapes

Similar Problems to Global Plus dangers of breaking up significant objects

Computational Simple Some Schemes seem to work well in practice

17

Regioning

• Break Image down into visually coherent areas

Can identify meaningful areas and objects

Computationally intensive Unreliable

18

Color

• Produce a color signature for region/whole image

• Typically done using color correllograms or color histograms

19

Color Features• Color Histograms

– Color Space Selection– Color Space Quantization– Color Histogram Calculation– Feature Indexing– Similarity Measures

• Color Layout– Histograms based on spatial distribution of single color– Histograms based on spatial distribution of color pair– Histograms based on spatial distribution of color triple

• Other Color Features– Color Moments– Color Sets

20

Color Space Selection

• Which Color Space?– RGB, CMY, YCrCb, CIE, YIQ, HLS, …

• HSV?– Designed to be similar to human perception

21

HSV Color Space

• H (Hue)– Dominant color (spectral)

• S (Saturation)– Amount of white

• V (Value)– Brightness

How to Use This?

22

Content Based Image Retrieval

• CBIR– utilizes unique features (shape, color, texture)

of images

Users prefer– To retrieve relevant image by semantic

categories

– But, CBIR can not capture high-level semantics in user’s mind

23

Relevance Feedback

• Relevance Feedback– Learns the associations between high-

level semantics and low-level features

• Relevance Feedback Phase1. User identifies relevant images within the

returned set2. System utilizes user feedback in the next round

To modify the query (to retrieve better results)3. This process repeats until user is satisfied

24

1st iteration

UserFeedback

Display

2nd iteration

Display

UserFeedback

Estimation &Display selection

Feedbackto system

25

Now, We have many features (too many?)

• How to express visual “similarity” with these features?

26

Visual Similarity ?

• “Similarity” is Subjective and Context-dependent.

• “Similarity” is High-level Concept.– Cars, Flowers, …

• But, our features are Low-level features.– Semantic Gap!

27

Which features are most important?

• Not all features are always important.• “Similarity” measure is always changing• The system has to weight features on the fly.

How ?

28

Q & A