8/13/2019 Angular Partioning

1/6

University of Wollongong

Research Online

Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences

2003

Sketch-based image retrieval using angularpartitioning

A. ChalechaleUniversity of Wollongong, uow_chalechale@uow.edu.au

G. NaghdyUniversity of Wollongong, golshah@uow.edu.au

A. MertinsUniversity of Oldenburg, Germany

Research Online is the open access institutional repository for the

University of Wollongong. For further information contact the UOW

Library: research-pubs@uow.edu.au

Publication Detailsis article was originally published as: Chalechale, A, Naghdy, G & Mertins, A, Sketch-based image retrieval using angularpartitioning, Proceedings of the 3rd IEEE International Symposium on Signal Processing and Information Technology (ISSPIT2003), 14-17 December 2003, 668-671. Copyright IEEE 2003.

http://ro.uow.edu.au/http://ro.uow.edu.au/infopapershttp://ro.uow.edu.au/eishttp://ro.uow.edu.au/http://ro.uow.edu.au/eishttp://ro.uow.edu.au/infopapershttp://ro.uow.edu.au/http://ro.uow.edu.au/http://ro.uow.edu.au/

8/13/2019 Angular Partioning

2/6

Sketch-based image retrieval using angular partitioning

Abstract

is paper presents a novel approach for sketch-based image retrieval based on low-level features. e

approach enables measuring the similarity between a full color image and a simple black and white sketchedquery and needs no cost intensive image segmentation. e proposed method can cope with imagescontaining several complex objects in an inhomogeneous background. Abstract images are obtained usingstrong edges of the model image and thinned outline of the sketched image. Angular-spatial distribution ofpixels in the abstract images is then employed to extract new compact and eective features using Fouriertransform. e extracted features are scale and rotation invariant and robust against translation. A collection ofpaintings and sketches (ART BANK) is used for testing the proposed method. e results are compared withthree other well-known approaches within the literature. Experimental results show signicant improvementin the recall ratio using the new features.

Disciplines

Physical Sciences and Mathematics

Publication Details

is article was originally published as: Chalechale, A, Naghdy, G & Mertins, A, Sketch-based image retrievalusing angular partitioning, Proceedings of the 3rd IEEE International Symposium on Signal Processing andInformation Technology (ISSPIT 2003), 14-17 December 2003, 668-671. Copyright IEEE 2003.

is conference paper is available at Research Online: hp://ro.uow.edu.au/infopapers/55

http://ro.uow.edu.au/infopapers/55http://ro.uow.edu.au/infopapers/55

8/13/2019 Angular Partioning

3/6

SKETCH-BASED IMAGE RETRIEVAL USING ANGULAR PARTITIONINGAbdo lah Chalechale, Golshah Naghdq

University of WollongomgWollongong, Australia

ABSTRACTThis paper presents a novel approach for sketch-based image re-trieval based on low-level features. The approach enables m easur-ing the similarity between a full color image and a simple blackand white sketched query and needs no cost intensive image seg-mentation. The proposed method can cope with images containingseveral complex objects in an inhomogeneous background. Ab-stract images are obtained using strong edges of the model imageand thinned outline of the skelched image. Angular-spatial dis-tribution of pixels in the abstract images is then employed to ex-tract new compact and effective features using Fourier transform.The extracted features are scale and rotation invariant and robustagainst translation. A collection of paintings and sketches (AR TBANK) is used for testing the proposed method. The results arecompared with three other well-known approaches within the lit-erature.Experimental results show significant improvement in the Re-call ratio using the new features.

1. INTRODUCTIONThe expeditious growth of digital imaging technology has resultedin the availability of a constantly increasing number of digital im-ages. Ima ge retrieval from multimedia databases is still an openproblem. Traditional textual methods have been shown to he inef-ficient and insufficient for searching in visual data. Con sequently,image and video content-b ased indexing and retrieval methods haveattracted many new researchers i n recent years [ I , 21.

Existing content-based image retrieval systems include QBIC131 PicSOM [4], MetaSEEk and VisualSEEk [S , 61. MPEG-7standard defin es descriptors driven from three main ima ge con-tent features, i.e. color, texture and shape L7 81 and VisualSEEk isusing object layout as an othe r content key [6].

User interaction is one of the most important aspects of anymultim edia system. A simple user interface makes the systemmore attractive and applicable. In sketch-based image retrieval,when the query example is a rough and simple black and whitedraft, hand drawn by the user, color and texture lose their originalability to serve as conte nt keys. Visual shape descriptors are usefulin sketch-based image retrieval when the model and the query im-age contain only on e object in a plain background [91. In multiple-object scen es, object lay out is a powerful tool. but object extractionand segmentation costs and rotation variance are the drawbacks.We focus our discussion on the problem of finding image fea-tures, invariant to scale, translation and rotation, which can e usedefficiently in sketch-based retrieval where the images have severalobjects in an inhom ogeneou s background. To our knowledge, thework of Hirata and Kato, Query by V isual Example (QVE) [ I O] is

Alfred MertinsUniversity of OldenburgOldenburg, Germany

the only one that addressed this problem previously. IBM adopts amodified version of the approach in the QBIC system [31. In thisapproach the query and the target images are resized to 64 x 64pixels first and then measuring the similarity is performed using ablock correlation scheme. The approach does not allow indexingand is computationally expensive. Although the method c an tol-erate small local rotations, it is not rotation invariant and does notallow for large global rotations.The Edge Histogram Descriptor (EHD) was proposed in thevisual parl of the MPE G-7 standard [I I] . It consists of a ISO-binhistogram . A given image is divided into 16 sub-images (4 x 4)

first. Local edge histograms are then computed for each sub-imagebased on the strength of five different edges (vertical, horizontal.45 diagonal, 135 diagonal, and isotropic). The E H D s basicallyintended fo r gray-image to gray-image matching but chang ing theinternal parameter Th ,d , , , a threshold f or edge extraction. couldmake it applicable for black and white queries in sketch-based re-trieval.T h e 2-D Fourier transform i n polar coordinates is employedfor shape description in [l2]. Its supremacy over I-D Fourierdescriptors, curvature scale space descriptor (MPEG-7 contour-based shape descriptor) and Zemike moments (MPEG-7 region-base d descr ipto r) has been show n in 1131.

borhood structure of the edge pixels to make an extended featurevector [14]. The vector is used efficiently for measuring the simi-larity between sketched queries and arbitrary model images. Thesemanti c power of the method is examined in [IS] Although themethod is scale and translation invariant it does not exhibit rotationinvariance property.In this paper we present a novel approach of feature extractionfor matching purposes based on angular partitioning of an abstractimage. Abstract images are obtained from the model image andfrom the query im age by two different procedures. The angular-spatial distribution of pixels in the abstract image is then employedas the key concept for feature extraction using Fourier transform.The extracted features are scale and rotation invariant and robustagainst translation. The major contribution of the paper is in ro-tation invariance and translation robustness propenies as. basedon the existing literature, there is no rotation and translation in-variant approach dealing with arbitrary images containing multi-ple objects while using sketched queries. The proposed algorithmand three othe r well-known approaches are implemented and ex-amine d using an art-image database (ART BANK). Experimentalresults show significant improvement in the Recall ratio using thenew approach.

Th e details of the proposed approach are discussed in the nextsection. Section 3 exhibits experimental results and evaluation.Section 4 conclude s the paper and poses s ome new directions.

Edg e Pixel Neighborhood Information (EPNI) is utilizing neigh-

668

8/13/2019 Angular Partioning

4/6

2. ANGULAR PARTITIONING OF ABSTRAC T. MAGE(APAI)The main objective of the proposed approach is to transform theimage data into a new structure that supports m easuring the simi-larity between a full color image an d a black and white hand-drawnsimple sketch.The edge map of an image carries the solid structure of theimage independent of the color attribute. Edg es are also provento be a fundamental primitive of an image for the preservationof both semantics and perceived attributes [16] Furthermore, insketch-based image retrieval, edges form the most useful featurefor matching purposes 114 9 171.According to the assumption that sketched queries are moresimilar to the edge maps which contain only the perceptive andvigorous edges, we obtain two abstr act images through the strongedges of the model image and the thinned version of the queryimage. Th e proposed features are then extracted fro m the abstractimages.The full color model imag e is initially converted to a gray in-tensity image I,by eliminating the h ue and saturation while retain-ing the luminance. The edges are then extracted using the Cannyoperator with U = 1 and Gaussian mask of size = 9 using thefollowing procedure for depicting the most perceived edges.The values of high and low thresholds for the m agnitude ofth epotential edge points are automatically computed in such a waythat only the strong edge s remain. T hi s improves the general sem-blance o fth e resulted edge map and t he hand drawn query. In orderto depict strong edges, let G be the Gaussian I-D tilter and let gbe the derivative of the Gaussian used in the Ca nny edge operator.Then

H k)= G ( i ) g ( k )is the I-D convolution of the Gaussia n and its derivative. The ma-trices N

X ( m _ n ) [ x I ( m , j ) H ( n - j ) ] j=1nr

Y ( m . n ) = I i , n ) H m )i = lf o r m = 1 , 2 , 3 , . . and n = 1 2 3 _ . N are the vertical andhorizontal central edge map s respectively, where is the numberof rows and N is the number of columns of the image I. Themagnitude of the edge points is then obtained as

r (m,n)=/ X ( m l r t ) * + ~ ( m . n ) 2 .For efficient selection of the high and low thresholds. we makea @-bin cumulative histogram of the r ( m , n ) alues and find theminimum index L in this cumulative histogram that is grater than

U * N , where a denotes the percentage of on edge pointsin the image a = 0.7 is an adequate choice for many images).To retain strong edges of the abstract image, 8 * L is selected asthe high threshold value and 0.40 * L is used for the low thresholdvalue in the Canny edge operator. p is a parameter that controlsthe degree of the strength of the edge points. High er /3s lead tolower number of edge points but mor e perceptive one s (see FigureI . Consequently, the gray image I is converted to an edge imageP using the Canny edge operator exploiting the above automaticextracted thresholds.For the query images, the procedure of black and white mor-phological thinning 181 is applied to extract a thinned version of

Figure : The effect of the p parameter on the edge maps

Figure 2: Angular partitioning splits the abstract image into Ksuccessive slices.the sketched image. This image, namely Q, shows an outline ofthe query and contain s the main structure of the user request. Itconta ins the spatial distribution of pixels similar to the strong edgemap of the model image P. The area in the bounding box of thecomparable P and Q images are normalized to J x J pixels us-ing nearest neighbor interpo1ation.The proposed norm alization ofthe P and Q images ensures the scale invariance and translationrobustness properties. Th e resulted image is called a b s l r a n imagen. We then define angular partitions (slices) in the surroundingcircle of the abstract image n see Figure 2). The angel betweenadjacent slices is = 2?r/K where K is the number of angularpartitions in the abstract image. K can be adjusted to achieve hi-erarchical coarse to fine representation requirement. Any rotationof a given image, with respect to its center, moves a pixel at sliceS, to a new position at slice S, where j = i + A mod K, fori . X = 0,1 ,2- . K 1. Figure 3 shows an example of abstractimage n nd its angul ar partitions.The num ber of edge points in each slice of R is chosen to rep-resent the slice feature. T he scale and translation invariant imagefeature is then { f ( i ) } where

W Rf i )= C R A 0 )

= e n p=oTfor = O; 1 , 2 . . K I. R is the radius of the surroundingcircle of the abstract image. Th e feature extracted above will becircularly shifted when the image Q is rotated T = 12a/K radian1 = 0: 1 , 2 . . ). To show this. let n, denote the abstract image nafter rotation by T radians in counterclockwise direction:

n , ( P ,B ) = n P , @ - T ) .

669

8/13/2019 Angular Partioning

5/6

Figure 3: An example of abstract image 12 and the angular parti-tions (slices) used for the feature extraclion.

are the imag e feature elements of the lr for the same i s . We canexpress f as

= f i - l )where i is a modulo M subtraction. f?( i )= f ( i in-dicates that there is a circular shift in the image feature { f T i ) }regarding to the image feature {f i ) } which representing 12 andil respectively.

Using I -D discrete Fourier transform of f ( i ) nd f T i )weobtainqU +c, ;1( i ) e - j z r i / KF ~ u ) +E:; f T ( i ) e ~ ~ Z K

= +E, ; f i e - 3 Z * ' / KcK-1-1 f ( . -jZau(i+l)/K

e - , 2 r u 1 / K ~ 2K , = - I % ) e

1.Based on the property lIF u)ll = llFT ~)ll,he scale and rota-tion invan ant image features are chose n as { l l F ~ ) l l }or U =0: 1;2 . K 1. The features are also robust against translationbecause of the proposed normalization of the abstract images.

The similarity between the model and the sketched images ismeasured by the e (Manhattan) distance between the two featurevectors. Experimental results (Section 3) confirm the robustnessand efficiency of the method.

3. EXPER IMENT AL RESULTS AND EVALUATIONThis section presents experimental results using the new proposedapproach in comparison with three other methods known from theliterature. We made a collection of different model and query im-ages called AR T BANK. Currently, it contains 365 full color im-ages of various sizes in the model part and 180 sketches in its querypart. Ima ges in the model part are mostly art works gained fromthe World A n Kiosk at California State University. Images in the

query pa n are black & white images which are draft sketches simi-lar to 5 arbitrary candidates from the model part and their rotatedversions (go', 180 and 270 ).The ART BANK was used as the test bed for the followingfour approaches. The proposed method (APAI), the QV E approach ,as it used in the QB lC system [3], MPEG-7 Edge Histogram De-scriptor (EHD) [?], and polar Fourier descriptor (PFD) appr oach[131. All the methods were tested using the sam e database.In the APAI method (Section 2). we applied 3, J = 129and = So esulting in a 72-entry feature vector f . In the EHDmethod, desiredxnumafdlockswas set to I100 and Th,d , , set to11(the default values) for the model im ages and Th,d,, was se t 0for the queries as they are binary images. A 150-bin histogram wasobtained in this approach employing local, semi-global and globalhistograms. We followed the algorithm given in [I31 to obtain a60-bin feature vector in PFD approach. The quantization stagein the EHD method was ignored to put all methods in the sam eSiNation.The queries were depicted from the following sets (each con-tains 45 images): the original sketches Qo). their 90(Qso), 180(Qmo) and the 270 (Q270) rotated versions. This is to simula tedifferent vertical and horizontal directions when posing a sketchedquery to the retrieval system.

Recall ratio R,, I O ] was used for the evaluation of retrievalperformance. It shows the ratio to retrieve the original full colormodel image in the best n-candidates. That isqueries retrieved the w e t mage in the first retrievals

total number of queries, = * 100The R, was obtained for each approach using the four differ-entq uery sets. Figure 4exhibits the retrieval performance supremacyof the proposed APAI method over the other approaches. For thequeries with the same directions as the model images Qo set). theretrieval performance of QVE, PFD a nd EH D methods de cline re-spectively compared to the APAl method for almost all n's. TheRecall ratio R, of the PFD method is higher than QV E and EHD

in all rotated cases. QVE and EHD m ethods show missing rota-tion invariance while PFD method is more robust with respect torotation and the APAl exhibits the best rotation invariant property.The maximum R , of the PFD method is 95% for n = 100 in setQISOut R,, f the APAl reaches to 100% for n 2 SO in all sets.

4 C O N C L U S I O N SThe approach presented in this paper (APAI) enables measuringthe similarity between a full color model image and a sim ple blackwhite sketched query. The images are arbitrary and may con-tain several complex objects in inhomogeneous backgrounds. Theapproach deals directly with the whole image and needs no costintensive image segmentation and object extraction. Ab strac t im-ages were defined based on the strong edges of the model imageand the morphological thinned outline of the query image. An-gular partitioning of the abstract image, using Fourier transform.is exploited to exuact features that are scale and rotation invariantand robust against translation. Experim ental results, using APAIapproach and the ART BANK as the test bed, show significant im -provement in the Recall ratio over three other approaches knownfrom the literature.The parlitioning scheme could be refined to improve retrievalperformance. We also intend to investigate sub-image search bydividing the image into several regions and applying the approachto each region successively.

670

8/13/2019 Angular Partioning

6/6

Figure4: R, verses n for sets Qo. Q90, Qlso and Q m espectively.Acknowledgments. The authors would like to acknowledge theWorld Art Kiosk at Califomia State University for providing thert works used in this study. We also thank Dr. Nargess Yasini.Andrew Mani and Khadijeh Moosavian who kindly help us pro-ducing sketched queries.

5. REFERENCES[I] .W.M. Smeulders, M. Worring. S. Santini, A. Gupta, andR. lain, Content-b ased image retrieval at the end of theearly years. IEEE Trans. on Patr.Anal. and Mach. Inrell.,vol. 22, no. I pp. 1349-1380.2000.121 Y. Rui. T.S. Hu ang , and S.F. Chang, Image retrieval: currenttechniques, promising directions, and open issues, Joumal

of Visual Commun andlmageRepresentarion,vol. 10, no. 1,pp. 39-62, 1999.131 W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasm an,D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin, TheQBIC project: querying images by content using color, tex-ture, and shape, in Proc. ofSpie, USA, 1993, vol. 1908, pp.173-187.[41 J. Laaksonen, M Koskela, S. Laakso. and E. Oja. PicSO M-content-based image retrieval with self-organizing maps,

Patr Recogniriun Letters, vol. 21, no. 13-14, pp. 1199-1207,Dec. 2000.

151 M Beigi, A. B Benitez, and S . Chang, Meta SEEk acontent-bas ed meta-sea rch engine for images. in Pmc. ofSpie ,USA, 1997 , vol. 33 12 .p ~ . 18-128.161 J. R Smith and S. Chang, VisualSEEk: a fully automatedcontent-based image query system, in Pm r. ACM M ulrime-dia 96..NY, USA , 1 996, pp. 87-98.[7] B.S. Manjunath. J:R. Ohm . and V. V. Vasudevan. Colo rand texture descriptors, IEEE Trans. on Circ. and Sysr.forVideo Technology, vol. 11. no. 6. pp. 703-715, June 2001.[8J M Bober, Mpeg-7 visual shape descriptors, IEEE Trans.on Circ. and Syst. for Video Technology, vol. 1I , no. 6 , pp.716-719. June 2001.

191 H. H. S. Ip, A. K. Y. Cheng, W. Y. E Wong, and 1 Feng,Affine-invariant sketch-based retrieval of images, in p m c .IEEE Inr. conf on Compur. Graphics, 2001, p p. 5 5 4 1 .[ I O ] K. Hirata and T.Kato, Query by visual example-contentbased ima ge retrieval: i n Advances in Database Technology- EDBT 92. Berlin, Germany, 1992. pp. 56-71[I ] ISO/IEC JTCI/SC 29/WG I UN4063, MPEG -7 Visual panof experim entation Model Version 10.0. Singapore, March1121 D. Zhang and L. Guojun. Generic fourier descriptor forshape-based image retrieval. in Pmc. IEEE Inr. Conf onMulrimedia and Erpo, 002, vol I pp. 425428 .[I31 D. Zhang and L. Guojun , Shape-ba sed image retrieval us-ing generic fourier descriptor: Signal Pmcessing: Image

Commun.. vol. 17, no. IO, pp. 82.5-848.2002.[I41 A. Chalecha le and A. Mertins, An abstract image represen-tation based on edge pixel neighborhood inform. (EPNI), inLecture N otes in Compur. Science, EurAsian-ICT 2W2:ln-form. and Commun. technology 2002, vol. 2510, pp. 67-74.[IS] A. Chalechale and A. Merlins, Semantic evaluation and ef-ficiency compa rison of the edge pixel neighboring histogramin image retrieval, in WITSPO2firsr wo rkh op on the in-

ternet, relecummunicarions and signal pmcess ing, Australia,1161 L. Atzori and F.G.B. De Natale, Error concealment invideo transmission over packet networks by a sketch-basedapproach: Signal Processing: Image Commun.. vol. 15. no.

1-2,pp. 57-76, 1999.1171 S. Matusiak, M. Daoudi, T. Blu, and 0. Avaro, Sketch-based images database retrieval: in Advances in Mulfimediu

Inform. Sysr. 4rh Inr. Workshop, MIS98. Pmc. , 1998.[I81 R. C. Gonzalez and R.E. Woods, Digital Image Processing,Addison-Wesley, 1992.

2001.

2002, pp. 179-184.

671