Integrating Color And Spatial Information for CBIR

Integrating Color And Integrating Color And Spatial Information for Spatial Information for

CBIRCBIR

NTUT CSIE D.W. Lin2003.8.26

References

L. Cinque, G. Ciocca, S. Levialdi, A. Pellicano, and R. Schettini, “Color-based image retrieval using spatial-chromatic histograms,” Image and Vision Computing, 19 (2001) 979-986

M.S. Kankanhalli, B.M. Mehtre, and H.Y. Huang, “Color and spatial feature for content-based image retrieval,” Pattern Recognition Letters, 20 (1999) 109-118

S. Berretti, A.D. Bimbo, and E. Vicario, “Spatial arrangement of color in retrieval by visual similarity,” Pattern Recognition 35(2002) 1661-1674

The representation of color

Color histogram– Global color histogram– Global color histogram + spatial info.– (fixed) Partition + local color histogram

Dominant color– Extracting the representative colors of image

via VQ or clustering (e.g. k-means algorithm)– Spatial info. can be attained

• Histogram refinement• Specific-color pixel distribution (single, pair, triple …)• Edge histogram …

Non-adaptive

Spatial-chromatic histograms [1]

SCH – global color histogram with info. about (single) pixel distribution

SCH attempts to answer:– How many meaningful colors? color space

quantization– Where the pixels having the same color?

location of region(with same color)– How are these pixels spatially arranged?

distribution of region

SCH – Color representation

Color representation– CIELAB Munsell ISCC-NBS

CIELAB– CIEXYZ CIELAB

CIELUV– a, b: opponent color

( green red, blue yellow )

SCH – Color representation

Munsell color system– Hue value chroma

SCH – Color representation

ISCC-NBS Centroid Color System– Partitioning the Munsell color system into 267

blocks, each blocks represented by an unique linguistic tag and the block centroid (Munsell coordinates)

Using back-propagation NN to transform – CIELAB Munsell ISCC-NBS

SCH – Feature vector

The definition of SCH for image ISI(k) = (hI(k), bI(k), σI(k))

– k: kth quantized color (1~c)– hI(k): pixel amount(ratio)– bI(k): baricenter (normalized mean coordinates)– σI(k): standard deviation of (spread)

Properties:– Insensitive to scale changes(via normalization)– Compact representation and rapidly computing

SCH – Similarity measure

Similarity function

– c: number of quantized color– d(·): Euclidean distance– : max. distance2

SCH – Effectiveness measure

– S: relevant items in DB– : retrieved set (short list) for a query– : relevant items in retrieved set

Precision if

Recall if

S

S

qq

qq

qq

qq

s

IE

EI

II

EI

RR

RR

RR

RR

IRq

IRq

ERq

Color and spatial clustering [2]

k-means algorithm– Iteration version– Two-pass version– VQ (LBG algorithm)

Proposed color clustering (two-pass)– Generating a new cluster while d(p, Ci) > T– Merging those clusters with small population

to the nearest cluster

Color and spatial clustering

Spatial clustering– Based on the clustered color layer– Using connected components labeling to

separate the spatial clusters– Discarding those clusters with small

population or lower density(embedded rectangular)

Feature vectors

For image I, color clusters can be givenCci = {Ri, Gi, Bi, λci, xci, yci} i: 1..m (number of color cluster)Ri, Gi, Bi: representative color of clusterλci: pixel ratio of cluster to totalxci, yci : centroid of cluster

fc={Cci|i=1, 2, …, m} Do the same to color-spatial clusters

Similarity measure

1: color distance between color cluster(RGB) 2: relative frequency of pixels of color cluster () 3: spatial distance between color cluster(x, y) 4: relative frequency of pixels of color-spatial cluster

() 5: spatial distance between color-spatial cluster(x, y)

5544332211),( IQD

Spatial arrangement of color[3]

The back-projection from dominant colors to the image results in an exceedingly complex model(e.g. [2])

Authors proposed a descriptor, called weighted walkthroughs, to capture the binary directional relationship of two complex sets of pixels

Weighted walkthroughs

– The model can be extended to represent the relationship of two sets A, B

– w11 evaluates the number of pixel pairs aA and bB such that b is upper right from a

+,+ w+1+1

aB

+, -

-, +

-, -

bbB

abjabiji yxyyCxxCB

Baw dd1,,

bbA B

aaabjabiji yxyxyyCxxCBA

BAw dddd1,,

Ci: characteristic function of negative/positive real number|B| : area of B i,j : ±1

Compositional computation

Reducing the region to a set of rectangular The weight between A and B1B2 can be derived

by linear combination of A/B1 and A/B2

Distance of WW

3 directional indexes:– wH(A, B) = w1, 1 (A, B) + w1, -1 (A, B)

– wV(A, B) = w-1, 1 (A, B) + w1, -1 (A, B)

– wD(A, B) = w-1, -1 (A, B) + w1, 1 (A, B)

Spatial distance

A B A

B

+,+ w+1+1

aB

+, -

-, +

-, -

A

B

DDVVHH wwwwwwwwDs ,

Arrangements comparing

Image model:< E, a, w >

E: set of spatial entities (color-clustered region)

a: E A { anya} (chromatic label)

w: E E W { anys} (spatial description)

Arrangements comparing

Distance between image model Q and D

: injective function(interpretation, association between query and model image)

– DA : chromatic distance (L*u*v*)– DS : spatial distance– Nq : number of entities in query

qN

k

k

hhkhks

Nq

kkkA qqqqDqqDDQ

1

1

11

,,,)1(,,

Future works

Finish the color-spatial study(geometric-enhanced histogram)

Study wavelet and JPEG 2000