Improving Medical Image Retrieval through Multi-Descriptor SimilarityFunctions and Association Rules

Renato Bueno1, Marcela X. Ribeiro1, Agma J. M. Traina2

and Caetano Traina Jr.2

1Department of Computer Science – DC, Federal University of Sao Carlos, SP, Brazil2Department of Computer Science – ICMC, University of Sao Paulo at Sao Carlos, SP, Brazil

{renato, marcela}@dc.ufscar.br {agma, caetano}@icmc.usp.br

Abstract

Content-based image retrieval (CBIR) systems still face

the problem of low precision of system results. To improve

the precision of such systems, many image visual extractors

have been developed and employed to represent the ima-

ges. However, the usage of a large number of extractors and

consequently, a large number of features, leads to the “di-

mensionality curse”, where the retrieval performance and

the query accuracy diminish. In this paper, we propose a

new method, called Statistical Fractal-scaled Product Me-

tric (SFPM), to maximize the accuracy of CBIR systems and

speedup similarity queries. The SFPM method combines

association rule mining and the Fractal-scaled Product Me-

tric (FPM) [4], to determine a reduced set of features and

appropriate scale factors in multi-descriptor image simila-

rity assessment. The FPM is an unsupervised method to

determine a scale factor among features in multi-descriptor

image similarity assessment based on the Fractal Theory.

Experiments have shown that SFPM reduced the feature

vector size in up to 65% and improved in up to 27% the

query precision when comparing with the use of the FPM te-

chnique. The results show that the proposed method SFPM

is effective in determining a reduced set of features and a

near-optimal set of scale factors for the descriptors invol-

ved, and it is well-suited to improve the quality of content-

based query in CBIR systems.

1. Introduction

The Digital Imaging and Communications in Medicine(DICOM) standard allows storing the study informationwith the image. In this context, the development of picture

archiving and communication system (PACS) has contribu-ted to the effective use of digital images in diagnosing aswell as training new physicians. In order to be effectivelyuseful, image retrieval in PACS should be fast and consis-tent with the specialist expectation. Hence, content-basedimage retrieval (CBIR) techniques have been intensively re-searched in the last years.

Most queries in a PACS are based on metadata. Howe-ver, querying images based on their visual characteristicshas proven to be a suitable complement to text-based se-arch. For instance, visual features not only allow the retrie-val of cases where patients have similar diagnoses, but alsothe identification of cases with visual similarity but differentdiagnoses [13]. Content-Based Image Retrieval (CBIR) isa technology that helps to manage digital picture archivesorganized by their visual content [9].

To obtain the visual representation of the images, theyare submitted to image processing algorithms, which gene-rate a mathematical signature describing the properties ofthe image content. Thereafter, the signatures can be com-pared through a function to measure the similarity betweenthem, yielding performing queries based on similarity cri-teria. One of major challenge of CBIR systems is the lowprecision of the systems results. One way of improving thequery accuracy is to employ various feature extractors todescribe the image content. However, uncontrolled incre-ase of the features into a composite vector may degrade theperformance and the precision of the results as well. Com-bining multiple descriptors requires a strategy that both ena-bles employing the best integration between the binomialfeature extractor-distance function for each descriptor, andstates a proper balance of the descriptors in the similaritycalculations. However, the methods for weighting multipledescriptors encountered in the literature are either based on

978-1-4244-9166-7/10/$26.00 ©2010 IEEE

exhaustive experimentation or in supervised techniques.

In this paper we present a new method to improve

the CBIR precision and speedup similarity queries. This

method combines association rule mining and the Fractal-scaled Product Metric (FPM), to determine a reduced set of

features and a set of scale factors in multi-descriptor image

similarity assessment. The association rule mining is em-

ployed to perform feature selection in each descriptor set,

and the FPM method is employed to balance these redu-

ced multi-descriptors. The FPM method is based on the

intrinsic dimensions of the feature spaces, approximated by

their respective fractal dimensions. Moreover, the compo-

site distance function enables using scalable access methods

to speed up data searching. Experiments have shown that

the proposed method is effective in determining a reduced

set of features and a near-optimal set of scale factors for the

descriptors involved and, also, the method always improves

the precision of the results.

The remainder of the paper is organized as follows. Sec-

tion 2 introduces the basic concepts for the paper and the

main related work. The proposed method is described in

Section 3.2. Section 4 shows results of some of the experi-

ments performed to evaluate the proposed method. Finally,

Section 5 presents the conclusions and future work.

2. Background and Related Work

In CBIR, image retrieval is mainly performed regarding

similarity. The basic types of similarity queries are the k-

nearest neighbor query and the range query. Similarity que-

ries are performed comparing the feature vectors using a

distance function - or dissimilarity function - to quantify

how close (or similar) two vectors are.

Several distance functions can be considered, e.g., the

Minkowski function family (including the well-known Eu-

clidean distance function), the Mahalanobis distance func-

tion and the cosine function. Two crucial issues in the CBIR

systems are: first, how to define a signature (feature vector)

that represents the image content; second, how to compute

the similarity between a pair of images based on their sig-

natures. [9].

The visual properties of an image are commonly descri-

bed in terms of color, texture and shape. These descriptors

are extracted by image processing algorithms either glo-

bally for the entire image or locally for regions of interest,

defined manually or by a segmentation algorithm. These

algorithms, called feature extractors, compute a number of

values, usually numeric, that compose the signatures repre-

senting the images, also referred to as feature vectors. There

are many image feature extractors documented in the litera-

ture, such as the histogram for color, the Haralick descrip-

tors for texture and the Zernike moments for shape [1].

In most real situations, employing various feature ex-

tractors to describe the image content improve the accuracy

of the results because they provide complementary charac-

teristics that contribute to the representation of the image

content. However, the usage of a large number of extrac-

tors and consequently, a large number of features, leads to

the “dimensionality curse”, where the retrieval speed and

the query accuracy diminish. Consequently, suitable appro-

aches to combine feature extractors without degrading the

results are needed.

A suitable approach to combine multiple descriptors

computes the similarity for each of them in a base level and

then aggregates the partial similarities using a composition

function. Let any image x be represented by a set of n des-

criptors x1, . . . , xn, each generated by a feature extraction

algorithm, and δ1, . . . , δn be distance functions defined over

the domains of the respective descriptors. The composition

distance function ∆ between two images x, y is a combi-

nation of the individual distances δi(xi, yi), usually given

by:

∆(x, y) =n�

i=1

wi · δi(xi, yi) (1)

where wi is the weight assigned to the respective descrip-

tor. Several systems employed this strategy [11, 7], but the

main drawback of such systems is that they determine the

weights of each component by means of exhaustive search.

Some approaches were proposed relying either on relevance

feedback cycles [10] or on supervised techniques[6]

Metric spaces are more adequate to represent multime-

dia data, as they only require the elements and their pairwise

distances [19]. A metric space is formally defined as a pair

�S, δ�, where S is a data domain and δ is a metric. A me-

tric is a distance function that satisfies the following metricaxioms, ∀x, y, z ∈ S:

1. symmetry: δ(x, y) = δ(y, x);

2. non-negativity: 0 < δ(x, y) < ∞ if x �= y and

δ(x, x) = 0;

3. triangular inequality: δ(x, y) ≤ δ(x, z) + δ(z, y).

Therefore, both adimensional1

and dimensional data can

be represented in a metric space, being sufficient to define

proper metrics to compare the data.

The usage of a metric distance to compute the dissimila-

rity between images also enables to adopt a Metric Access

Method (MAM) to speed up the similarity queries execu-

tion. There are many MAMs proposed [12], like M-Tree [8]

and Slim-tree [17]. Experimental evaluation has shown that

MAMs are very effective for indexing image data, achieving

superior performance and scalability than other structures.

1Adimensional data are those for whom no dimensionality can be as-

signed.

Having introduced the basic concepts, we present our

new method to improve CBIR systems combining multiple

extractors in the next section.

3. Proposed Method: SFPM

The proposed method Statistical Fractal-scaled ProductMetric (SFPM) is a technique that maximizes the query ac-

curacy in the same time that minimizes the number of featu-

res employed to perform the query. The method combines

a statistical approach to perform feature selection and an

approach based on fractal theory to find the most suitable

weight of features, according to their distinguishing power.

The SFPM method employs the Fractal-scaled Product Me-tric (FPM), wich is based on the intrinsic dimensions of the

feature spaces, approximated by their respective fractal di-

mensions. The composite weight set and the selected fea-

tures enable the use of scalable access methods to speed up

data searching.

The method SFPM is illustrated in Figure 1. The images

from the database are submitted to a set of feature extrac-

tors. Each image is represented by set of feature vectors

which are submitted to a feature selection process through

the StARMiner algorithm (as described in the next subsec-

tion), producing a reduced set of features for each extractor,

i.e. producing reduced feature vectors regarding their di-

mensionality. The properties of the reduced feature vectors

are analyzed using the fractal theory by the FPM method,

as described in Section 3.2, producing scale factors for the

multi-descriptors. Finally, the reduced feature vectors and

the scale factors are employed in the CBIR system to exe-

cute the similarity queries asked by the user, improving the

precision of the result and the speed of the query execution.

3.1 Statistical Association Rules for Fea-ture Selection

The SFPM method employs an algorithm, named StAR-

Miner [14], to mine statistical association rules from a da-

taset. The mined rules are used to select the most relevant

features, making a new and enhanced representation of the

images.

Let T be a dataset of medical images, x an image class,

Tx ∈ T the subset of images of class x and F a feature. Let

µF (Z) and σF (Z) be, respectively, the mean and standard

deviation of the values of feature F in the subset of ima-

ges Z. The algorithm uses three thresholds defined by the

user: ∆µmin - the minimum allowed difference between

the average of the feature F in images from class x and the

average of F in the remaining dataset; ∆σmax - the ma-

ximum standard deviation of F values allowed in a given

class and; γmin - the minimum confidence to reject the hy-

pothesis H0. StARMiner algorithm mines rules of the form

x → F , if the conditions given in Equations 2, 3 and 4 are

satisfied.

µf (Tx)− µf (T − Tx) ≥ ∆µmin (2)

σf (Tx) ≤ ∆σmax (3)

H0 : µf (Tx) = µf (T − Tx) (4)

In Equation 4, H0 should be rejected with a confidence

equal to or greater than γmin, in favor of the hypothesis that

the means µf (Tx) and µf (T − Tx) are statistically diffe-

rent. A rule x → F , returned by the algorithm, relates a

feature F with a class x, where values of F have a statisti-

cally different behavior in images of class x. This property

indicates that F is an interesting feature to distinguish ima-

ges of class x from the remaining images. Therefore, the

feature selection performed in our approach is: The set Sof relevant features are composed of the features F presentin the rules returned by StARMiner. These features have a

particular and uniform behavior in images of a given class.

3.2 Fractal-scaled Product Metric (TheFPM method)

Feature extractors produce signatures comparable th-

rough metrics, so each one generates a metric space. The

standard way of aggregating n metric spaces Mi = �Si, δi�,1 ≤ i ≤ n, is defining a metric over the cartesian product

M1 ×M2 × ...×Mn, which is called a product metric.

The Fractal-scaled Product Metric (FPM) method [4]

calculates the scale factors (the weights wi of Equation

1) between the composed metrics based on the correlation

fractal dimension D2 of each original metric space created

by the extracted descriptors. Knowing D2 of a dataset al-

lows predicting its properties as being similar to that of a

dimensional dataset with approximately the same embed-

ded dimension[3].

The main idea of the FPM is to identify the contribu-tion for the overall similarity calculation of each descriptor.

Since the value of the intrinsic dimension reveals the pre-

sence of correlations among the attributes of a dataset, the

fractal dimension provides an estimate of a lower bound for

the number of features needed in a similarity search to keep

the essential of data information.

Therefore, scaling the multiple descriptors in the pro-

duct metric by their intrinsic dimensions approximated by

the value of D2, neither overestimate nor underestimate the

contribution of each descriptor in the similarity assessment.

The correlation fractal dimension can be calculated using

the Box Counting method [15, 18] or using the technique

proposed in [16].

The FPM method performs normalization to reduce the

effects of the range difference in the similarity computation.

Vector 1

STARMiner

Medical Image Database

Extractor 1

Extractor 2

Extractor n

...

Vector 2

Vector n

...

Reduced Vector 1

Reduced Vector 2

Reduced Vector n

...

FPM

Reduced Vector 1 x Weight 1

...

Similarity Query Execution Reduced Vector 2 x Weight 2

Reduced Vector 3 x Weight n

USER

QueryExecution

Results

SFPM

Figure 1. Pipeline of the proposed method SFPM.

To do so, it uses the largest known distance dmaxi for eachdescriptor. In summary, the FPM metric is given by:

∆(x, y) =n�

i

D2i · δi(xi, yi)dmaxi

(5)

The values of dmaxi can be measured by computing thedistances between all pairs of elements considering eachdescriptor, but this operation is quadratic on the number ofelements. However, cheaper techniques can be employed toestimate this value. For example, if a MAM is indexing thedata referred to the descriptor i, it is feasible to get a closeapproximation stating dmaxi as the diameter of the regioncovered by the MAM’s root node. Refer to paper [5] to amore complete discussion of the notion of scaling metricspaces using fractals.

4. Experiments

This section presents results of experiments performedto evaluate the proposed method. The software developedwas implemented in C++ using the Slim-tree MAM [17],available in the Arboretum2 library to index data. To per-form the experiments we have used medical image datasetsprovided by the Clinical Hospital of the Medical School ofRibeirao Preto, University of Sao Paulo, Brazil.

The first dataset used, called MRI 704, consists of 704images of images of Magnetic Resonance Imaging (MRI)exams. The dataset is divided in 40 classes according tothe region of body examined, the slice position and the type

2Available at http://gbdi.icmc.usp.br/arboretum

of specified section. Each image of the MRI 704 databasewas processed by two feature extractors: Haralick’s Textureand Zernike moments, that contains the first 256 momentsof Zernike describing the image shapes.

The second dataset (2), called CT Lung ROIs, consistsof 3257 images, containing Regions of Interest (ROIs) ofimages of Computed Tomography (CT) lung exams, and itis organized in 6 classes, being one normal and 5 of abnor-mal patterns. These images were processed using the Ha-ralick’s Texture and the first 256 moments of Zernike. TheManhattan distance (L1) was used in all experiments.

Figure 2. Sample images of CT Lung ROIs

database.

4.1. Results

We employed Precision versus Recall (P×R) graphs [2]to evaluate the quality of the query results. The recall ofa query is calculated as: recall = |Ra|

|R| , where |Ra| is the

number of relevant elements retrieved and |R| is the numberof relevant elements that should be retrieved. The precisionof a query is given by: precision = |Ra|

|A| , where |Ra| is thenumber of relevant elements retrieved and |A| is the size ofthe answer set. A simple way to interpret a P×R curve isthe closer the curve to the top, the better the technique.

The first step of SFPM is the StARMiner feature selec-tion process, which performed a significant reduction in thenumber of features of the feature vectors. StARMiner mi-nes rules relating image classes and the most discrimina-tive features. The feature selection process performed byStARMiner removes the noise and redundant features thatinterferes negatively in the query result, not only speedingup the whole process, but also improving the query preci-sion. Regarding the MRI 704 dataset, the StARMiner se-lected 86 from the 256 original set of features generatedby Zernike extractor, and 90 from the 140 features gene-rated by the Haralick texture descriptor. When analyzingthe CT Lung ROIs, StARMiner selected 123 from the 256initial set of Zernike features, and 80 from the 140 initialtexture features. The feature selection step of SFPM per-formed a reduction of up to 65% in the feature vector size,making the content-based search faster and also more accu-rate.

Figure 3 shows the P×R graph comparing the retrie-val quality of employing each individual descriptor of theMRI 704 database, the combination of original descriptorsusing the FPM method, and the results obtained with theproposed method, the SFPM. The SFPM method outperfor-med the individual descriptors and the FPM method in allrecall levels, having improved in some of them the preci-sion up to 15% over using only the Zernike descriptor, upto 157% over using only the Haralick descriptor and up to14% over FPM method.

The Figure 4 shows the results of the same kind of expe-riment with the CT Lung ROIs database, showing the P×Rgraph of the invidual descriptors, FPM applied on originaldescriptors, and the results obtained with SFPM method.Again, the SFPM method achieved better results in all re-call levels, with precision improvements of up to 75% overusing only the Zernike descriptor, up to 47% over using onlythe Haralick descriptor and up to 27% over FPM method, insome recall levels.

5. Conclusions and Future Work

In this paper we proposed the Statistical Fractal-scaledProduct Metric (SFPM) method to improve the qualityof content-based query in CBIR systems. The proposedmethod combines association rule mining and the Fractal-scaled Product Metric (FPM) technique to determine a re-duced set of features and a set of scale factors for multi-descriptor image similarity evaluation.

Figure 3. P×R graph of each single descriptorof the MRI 704 database, of the FPM methodand the SFPM combining them.

The association rule mining is employed to perform fea-ture selection using statistical measures. The FPM methodemploys the fractal theory to approximate the intrinsic di-mensions of the feature spaces and to determine the weightof the feature vectors in the similarity measure.

Experiments have shown that the proposed method is ef-fective in determining a reduced set of features and a near-optimal set of scale factors for the descriptors involved. Theproposed SFPM method reduced the feature vector size inup to 65%, and improved the precision of the queries inup to 157% regarding the usage of only a feature extractor.Also, the SFPM method improves in up to 27% the queryprecision when comparing with the use of the FPM techni-que.

Future work includes testing different distance functionsand image descriptors with the proposed method, and tes-ting other product metrics to combine multiple descriptors.

Acknowledgments

This work has been supported by FAPESP (Sao PauloState Research Foundation), CNPq (National Council forScientific and Technological Development), CAPES (Bra-zilian Federal Funding Agency for Graduate Education Im-provement), STIC-AmSud and Microsoft Research.

References

[1] Y. A. Aslandogan and C. T. Yu. Techniques and systems forimage and video retrieval. IEEE Trans. on Knowl. and DataEng., 11(1):56–63, 1999.

Figure 4. P×R graph of each single descriptor

of the CT Lung ROIs database, of the combi-

nation using the FPM method and using the

SFPM.

[2] R. A. Baeza-Yates and B. A. Ribeiro-Neto. Modern Infor-

mation Retrieval. Addison-Wesley, Wokingham, UK, 1999.

[3] A. Belussi and C. Faloutsos. Self-spacial join selectivity es-

timation using fractal concepts. ACM Trans. on Inf. Systems,

16(2):161–201, 1998.

[4] R. Bueno, D. S. Kaster, A. A. Paterlini, A. J. M. Traina, and

C. T. Jr. Unsupervised scaling of multi-descriptor similarity

functions for medical image datasets. In CBMS, pages 1–8,

2009.

[5] R. Bueno, D. S. Kaster, A. J. M. Traina, and C. Traina Jr.

Time-aware similarity search: a metric-temporal represen-

tation for complex data. In SSTD, volume 5644 of LNCS,

pages 302–319, Aalborg, Denmark, 2009. Springer.

[6] B. Bustos, D. Keim, D. Saupe, T. Schreck, and D. Vranic.

Automatic selection and combination of descriptors for ef-

fective 3d similarity search. In Multimedia Software Engi-

neering, pages 514–521, Miami, FL, USA, 2004. IEEE.

[7] J. C. Caicedo, F. A. Gonzalez, E. Triana, and E. Romero.

Design of a medical image database with content-based re-

trieval capabilities. In Advances in Image and Video Tech-

nology, pages 919–931, Santiago, Chile, 2007. Springer.

[8] P. Ciaccia, M. Patella, and P. Zezula. M-tree: An efficient ac-

cess method for similarity search in metric spaces. In VLDB,

pages 426–435, Athens, Greece, 1997. Morgan Kaufmann.

[9] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval:

Ideas, influences, and trends of the new age. ACM Comput.

Surv., 40(2):1–60, 2008.

[10] C. D. Ferreira, R. da Silva Torres, M. A. Goncalves, and

W. Fan. Image retrieval with relevance feedback based on

genetic programming. In SBBD, pages 120–134, 2008.

[11] D. Heesch and S. Ruger. Combining features for content-

based sketch retrieval – a comparative evaluation of retrieval

performance. In In Proc. 24th BCS-IRSG European Collo-

quium on IR Research, pages 41–52. Springer, 2002.

[12] G. R. Hjaltason and H. Samet. Index-driven similarity se-

arch in metric spaces. ACM TODS, 21(4):517–580, 2003.

[13] H. Muller, N. Michoux, D. Bandon, and A. Geissbuhler. A

review of content-based image retrieval systems in medical

applications-clinical benefits and future directions. Int. Jour-

nal of Medical Informatics, 73(1):1–23, 2004.

[14] M. X. Ribeiro, A. G. R. Balan, J. C. Felipe, A. J. M. Traina,

and C. Traina Jr. Mining statistical association rules to select

the most relevant medical image features. In First Internati-

onal Workshop on Mining Complex Data (IEEE MCD’05),

pages 91–98, Houston, USA, 2005.

[15] M. Schroeder. Fractals, Chaos, Power Laws. W. H. Free-

man, New York, 6 edition, 1991.

[16] C. Traina Jr., A. J. M. Traina, and C. Faloutsos. Distance

exponent: a new concept for selectivity estimation in metric

trees. In ICDE, page 195, San Diego, CA, USA, 2000. IEEE.

[17] C. Traina Jr., A. J. M. Traina, C. Faloutsos, and B. Se-

eger. Fast indexing and visualization of metric datasets

using Slim-trees. IEEE Trans. on Knowl. and Data Eng.,

14(2):244–260, 2002.

[18] C. Traina Jr., A. J. M. Traina, L. Wu, and C. Faloutsos. Fast

feature selection using fractal dimension. In SBBD, pages

158–171, Joao Pessoa, PB, Brazil, 2000.

[19] P. Zezula, G. Amato, V. Dohnal, and M. Batko. Similarity

Search: The Metric Space Approach, volume 32 of Advan-

ces in Database Systems. Springer, 2006.