T.Sharon1

Internet Resources Discovery (IRD)

Introduction to MMIR

T.Sharon2

Contents

• Visual Information Retrieval (VIR)– Images– Video

• Video Information Retrieval (VIR)

• Music Information Retrieval (MIR)

T.Sharon3

Visual Information Retrieval

• Introduction

• VIR system

• VIR information domains

• Querying video

• Advanced topics

T.Sharon4

Introduction• What is VIR?

• Who needs it?

• Questions and problems

T.Sharon5

What is VIR?

Query

VIRSystem

VIR allows users toquery and retrieve visual information.

Queries will be doneaccording to informationcontent.

T.Sharon6

Who needs VIR?• Libraries

• Museums

• Scientific Archives

• Image Warehouses

T.Sharon7

Questions and Problems• How can we search visual information?

• How can visual and non-visual information can be searched together?

• Problems:– visual information is subjectively interpreted.– few representations: images, graphics, video,

animations, stereoscopic images.– requires substantial amount of resources.

Dog??

T.Sharon8

VIR System

• Architecture

• Query formulation

• Match and ranking

• Query answer

• Refinement (relevance feedback)

T.Sharon9

Architecture of VIR System

T.Sharon10

Query Formulation

• Query by example:– sketch an example– give an image

• Query by giving values to visual features:– % colors– texture– describe textually

but use visual tools to define values.

T.Sharon11

Query matching and ranking

• Similarity test• Using combination of features, for example:

– colors– texture– shape– motion– additional information

• Actions in feature space can be:– maximal distance– K nearest neighbors

........ .....

Feature 1

Feature 2

Feature 3

T.Sharon12

Query Answer

Thumbnails:

• Images – DC images

• Video– built from

selected DC images (key frames)

T.Sharon13

Query Refinement

• Using a result image from previous query.

• Launch a new query.

• Modifying a result image with an image processing tool to specify an an additional criteria.

• Changing relative weights of visual features and get a new ranking to the previous results.

T.Sharon14

VIR Information Domains

• Information domain

• Queries at Pixels Level– examples– problems

• Implementations– color– color complex– shape

T.Sharon15

Information Domains

• Metadata information– alphanumeric, database scheme

• Visual characteristic– contained in the object– achieved by using computational process, usually

image processing

T.Sharon16

Queries at Pixels Level - Examples

Find all objects for which the 100th to 200th pixels are orange (RGB=255,130,0).

Find all the images that have about the same color (certain RGB) in the central region (relative or absolute).

Find all images that are a shifted version of this particular image, in which the maximum allowable shift is D.

T.Sharon17

Queries at Pixels Level -Problems

Pixel queries are noise sensitivecouple of noise pixels can cause to discard a good

image.

Do not work on rotations.Changes in lightning and imaging conditions

effect pixels significantly and bias queries.

T.Sharon18

Implementations

• Pixels location combined with

• Database scheme built by humans

• Example techniques:– Color– Color complex– Texture– Shape

T.Sharon19

Color• Method:

– Color definition• Hue (color spectrum)• Saturation (gray)

– Calculate histograms

• Enables queries:– Find all images for which more than 30% is sky blue and

more than 25% is grass green– Sort histogram drawers, find 5 most frequent colors, find all

other images with these color features– Find all images far from this image only D

T.Sharon20

Color Complex• Method: Create histograms quad-tree:

• Calculate image histogram

• Divide image to quarters and calculate histogram for each quarter

• Continue recursively till 16x16 squares

• Enables queries:– Find images for which:

• more than 20% red-orange pixels in the right upper quarter

• more than 20% yellow pixels in the left upper quarter

• about 30% brown pixels in the bottom half of the picture

– Find all images with red patch in the middle and blue patch around.

T.Sharon21

Shape• Method:

– Suppose we have graphics collection (clip arts)• contain pure colors (little hue changes, no saturation)

– Divide image to color areas so that each area contains pixels with the same pure color

– Calculate features for each area:• color, area, elongation (sqrt(perimeter)/area),

centrality (distance of shape centroid to image center)

• Enables queries:– Find all images containing white squares in the center

– Find all images containing 2 blue circles close to the center

T.Sharon22

Examples: Existing Systems• SaFe http://disney.ctr.columbia.edu/SaFe/

• Virage http://www.virage.com/virdemo.html

• QBIC http://wwwqbic.almaden.ibm.com/ (stamps) download!

http://wwwqbic.almaden.ibm.com/cgi-bin/pcd-demo/drawpicker (photos)

• MetaSEEK http://mahler.ctr.columbia.edu:8080/cgi-bin/MetaSEEk_cate

• WebSEEK• VisualSEEK• MELDEX http://www.nzdl.org/cgi-bin/gw?

c=meldex&a=page&p=coltitle

T.Sharon23

SaFe

T.Sharon24

QBIC - Histogram Query

T.Sharon25

QBIC - Color Layout Query