ImageDatabaseIndexing usinga Combination of InvariantShapeand Color Descriptions

�

RonaldAlferezandYuan-FangWangComputerScienceDepartment

Universityof CaliforniaSantaBarbara,CA, 93106U.S.A.

Abstract Image and video library applicationsarebecomingincreasinglypopular. The increasingpopu-larity callsfor softwaretoolstohelptheuserqueryandretrieve databaseimagesefficiently and effectively. Inthis paper, we presenta technique which combinesshapeand color descriptorsfor invariant, within-a-class retrieval of imagesfrom digital libraries. Wedemonstratethetechniqueon a real databasecontain-ing airplane imagesof similar shapeand query im-agesthat appeardifferent from thosein the databasebecauseof lighting and perspective. We were able toachievea veryhigh retrieval rate.

Keywords: images,video,libraries,features

1 Intr oductionImageand video library applicationsare becom-ing increasinglypopularaswitnessedby many na-tionalandinternationalresearchinitiativesin theseareas,an explodingnumberof professionalmeet-ings devotedto image/video/multi-media, andtheemergency of commercialcompaniesand prod-ucts.Theadventof high-speednetworksandinex-pensive storagedeviceshasenabledthe construc-tion of large electronicimageandvideo archivesandgreatlyfacilitatedtheir accesson theInternet.In line with this, however, is theneedfor softwaretools to help the userqueryandretrieve databaseimagesefficiently andeffectively.

Queryingan imagelibrary canbe difficult andoneof themaindifficultiesliesin designingpower-ful featuresor descriptorsto representandorganizeimagesin a library. Many existing imagedatabaseindexing andretrieval systemsareonly capableofbetween-classesretrieval (e.g.,distinguishingfishfrom airplanes). However, thesesystemsdo not�Supportedin part by a grant from the NationalScience

Foundation,IRI-94-11330

allow theuserto retrieve imagesthataremorespe-cific. In otherwords, they areunableto performwithin-a-classretrieval (e.g.,distinguishingdiffer-ent typesof airplanesor differentspeciesof fish).This is becausetheaggregatefeaturesadoptedbymany currentsystems(such as color histogramsand low-orderedmoments)captureonly the gen-eralshapeof aclassandarenotdescriptiveenoughto distinguishobjectswithin aparticularclass.

The within-a-classretrieval problemis furthercomplicatedif query images depicting objects,thoughbelongingto theclassof interest,maylookdifferent due to non-essentialor incidentalenvi-ronmentchanges,suchasrigid-bodyor articulatedmotion,shapedeformation,andchangein illumi-nation and viewpoint. In this paper, we addresstheproblemof invariant,within-a-classretrieval ofimagesby usinga combinationof invariantshapeandcolor descriptors.By analyzingthe shapeoftheobject's contouraswell asthecolorandtexturecharacteristicsof the enclosedarea, informationfrom multiple sourcesis fusedfor a more robustdescriptionof an object's appearance.this placesour techniqueat an advantageover most currentapproachesthat exploit eithergeometricinforma-tion or color informationexclusively.

The analysisinvolvesprojectingthe shapeandcolor information onto basis functions of finite,local support (e.g., splinesand wavelets). Theprojectioncoefficients,in general,aresensitive tochangesinducedby rigid motion,shapedeforma-tion, and changein illumination and perspective.We derive expressionsby massagingthesesetsof projectioncoefficients to cancelout the envi-ronmentalfactorsto achieve invarianceof the de-scriptors. Basedon thesefeatures,we have con-ductedpreliminary experimentsto recognizedif-ferent types of airplanes(many of them havingverysimilarshape)undervaryingilluminationand

viewing conditionsandhave achievedgoodrecog-nition rates.We show that informationfusionhashelpedto improve the accuracy in retrieval andshapediscrimination.

2 TechnicalDescriptionIn this section,we presentthe theoreticalfoun-dation of our image-derived, invariant shapeandcolor features.Invariantfeaturesform a compact,intrinsic descriptionof anobject,andcanbeusedto designretrieval andindexing algorithmsthatarepotentially more efficient than, say, aspect-basedapproaches.

Thesearchfor invariantfeatures(e.g.,algebraicandprojective invariants)is a classicalprobleminmathematicsdatingbackto the18thcentury. Theneedfor invariantimagedescriptorshaslong beenrecognizedin computervision. Invariant featurescan be designedbasedon many different meth-odsandmadeinvariantto rigid-bodymotion,affineshapedeformation,sceneillumination, occlusion,andperspective projection.Invariantscanbecom-putedeither globally, such is the casein invari-antsbasedon momentsor Fourier transformco-efficients,or basedon localpropertiessuchascur-vatureandarc length. See[3, 4, 5] for survey anddiscussionon thesubjectof invariants.

As mentionedbefore,our invariantfeaturesarederived from a localized analysisof an object'sshapeandcolor. Thebasicideais to projectanob-ject'sexteriorcontouror interiorregionontolocal-izedbasessuchaswaveletsandsplines.Thecoeffi-cientsarethennormalizedto eliminatechangesin-ducedby non-essentialenvironmentalfactorssuchasviewpoint and illumination. We will illustratethemathematicalframeworksusinga specificsce-nariowhereinvariantsfor curvesaresought.Theparticularbasisfunctionsusedin the illustrationwill bethewaveletbasesandsplinefunctions.In-terestedreadersarereferredto [1] for moredetails.

Several implementationissuesarisein this in-variant framework which we will briefly discussbeforedescribingthe invariant expressionsthem-selves.1

1. How are contoursextracted?

1A word on the notationalconvention: matricesandvec-tors will be representedby bold-facecharacterswhile scalarquantitiesby plain-face characters. 2D quantitieswill bein small letterswhile 3D and higher-dimensionalquantitiesin capital letters. For example,coordinates(bold for vectorquantities)of a 2D curve (small letter for 2D quantities)willbedenotedby � .

Or statedslightly differently, how is the problemof segmentation(separatingobjects from back-ground)addressed?Segmentationturnsout to bean extremelydifficult problemand,asfundamen-tal a problemassegmentationis, thereis no fail-proof solution. A “perfect” segmentationschemeis like theholy grail of low-level computervisionanda panaceato many high level vision problemsaswell.

We arenot in searchof this holy-grail, which,we believe, is untenablein the foreseeablefuture.In an imagedatabaseapplication,the problemofobjectsegmentationis simplifiedbecause� Databaseimagescan usually be acquiredun-der standardimagingconditionswhich allow theingest and catalog operationsto be automatedor semi-automated.For example, to constructadatabaseof airplaneimages,many bookson civiland military aircraftsare available with standardfront, side,andtop views taken againsta uniformor unclutteredbackground. (The above is alsotruefor applicationsin botany andmarinebiology.)Thisallowsthecontoursof theobjectsof interesttobeextractedautomaticallyor with theaid of stan-dardtoolssuchasthefloodfill maskin Photoshop.Furthermore,thecatalogingoperationsareusuallydoneoff-line anddoneonly once.Hence,a semi-automatedschemewill suffice.� On the other hand, query imagesare usuallytaken underdifferent lighting andviewing condi-tions. Objectsof interestcanbeembeddeddeeplyin clutteredbackgroundwhich makestheir extrac-tion difficult. However, we can enlist the helpof the user to specify the object of interest in-steadof askingthe systemto attemptthe impos-sibletaskof automatedsegmentation.A query-by-sketchor a“human-in-the-loop”typesolutionwithaneasy-to-usegraphicsinterfaceandsegmentationaids suchas the flood fill maskis perfectly ade-quatehereanddoesnot imposeundueburdenontheuser. This proved to be feasiblein our experi-ments.2. How are contoursparameterized?For a contourbaseddescription,a commonframeof referenceisusuallyneededthatallowspointcor-respondencesto be establishedbetweentwo con-toursfor comparison.Thecommonframeof refer-encecomprisesa commonstartingpoint of traver-sal,acommondirectionof traversal,andaparame-terizationschemethattraversesto thecorrespond-ing pointsin the two contoursat thesameparam-eter setting. We will first discussthe parameter-

ization�

issueand thenaddressthe issuesof pointcorrespondenceandtraversaldirection.

When defining a parameterizedcurve �����

�� � ��������������� , mostprefertheuseof theintrinsicarclengthparameterbecauseof its simplicity andthefactthatit is eitherinvariantor transformslinearlyin rigid-body motion anduniform scaling. How-ever, undermore generalscenarioswhereshapedeformationis allowed(e.g.,deformationinducedin an oblique view), intrinsic arc length parame-ter is no longer invariant. Suchdeformationcanstretchandcompressdifferentportionsof of anob-ject's shape,and a parameterizationbasedon in-trinsic arc lengthwill resultin wrongpoint corre-spondence.

It is well known that many shapedeformationanddistortionresultingfrom imagingcanbemod-eledasanaffine transform,throughwhich the in-trinsic arc length is nonlinearly transformed[2].An alternative parameterizationis thus required.There are at least two possibilities. The first,called affine arc length, is defined[2] as: ������� �! "�$#�&% #� "�(')� where "� � "� are the first and #� � #�arethe secondderivativeswith respectto any pa-rameter � (possibly the intrinsic arc length), and�+*,�.-/ is thepathalongasegmentof thecurve.

Another possibility [2] is to use the enclosedareaparameter: 01�324 ����65 � "�7%� "� 5 ')�98 Onecaninterprettheenclosedareaparameterastheareaofthetriangularregion enclosedby thetwo line seg-mentsfrom thecentroidof anobjectto two points* and - on thecontour. It canbeshown thatboththeseparameterstransformlinearly undera gen-eral affine transform[2]. Hence,they caneasilybemadeabsolutelyinvariantby normalizingthemwith respectto thetotalaffinearclengthor thetotalenclosedareaof the whole contour, respectively.Weusetheseparameterizationsin ourexperiments.3. How are identical traversal dir ection andstarting point guaranteed?It will be shown that the invariant signatures (tobedefinedlater)of two contoursarephase-shiftedversionsof eachotherwhenonly thestartingpointof traversaldiffers.Furthermore,thesamecontourparameterizedin oppositedirectionsproducesin-variantsignaturesthatareflippedandinvertedim-agesof eachother. Hence,a matchcanbechosenthat maximizescertaincross-correlationrelationsbetweenthetwo signatures.

Allowing an arbitrary changeof origin andtraversal direction, togetherwith the use of anaffine invariant parameterization,imply that no

point correspondenceis required in computingour invariants.

Now wearereadyto introducetheinvariantex-pressionsthemselves.Our invariantsframework isvery generalandconsidersvariationin anobject'simageinducedby rigid-bodymotion,affine defor-mation,andchangesin parameterization,sceneil-lumination,andviewpoint. Eachformulationcanbeusedalone,or in conjunctionwith others.Dueto the pagelimitation, we can only give a briefdiscussionof the invariantsunderrigid-body andaffine transformand summarizethe invariant ex-pressionsunderchangeof illumination andview-point. Interestedreadersare referredto [1] formoredetails.

Invariants under Rigid-Body Motion and AffineTransform Consider a 2D curve, ����� �� � ������������� � where � denotesa parameterizationwhich is invariant underaffine transform(as de-scribedabove), andits expansionontothewaveletbasis: � �9@+A �� (where > ��� is themotherwavelet)as B �C; � � � �����D: �

Scenarios Invariantexpressions

Rigid-bodymotion(usingsplinebasis)5 ucv9w x A ucy�w zc55 u|{Dw } A uc~.w 5

Affine transform(usingwaveletbasis)

ppG�V ,. pppp ,9 ,c ppAffine transform(usingsplinebasis)

pppp V . 9 pppppppp �c 3�| pppp

Perspective transform�@ �+���%aN| + ; ����

4 ']�(usingrationalsplinebasis ) where��� is theobservedimagecurve,and ; ��� is thedatabasecurve in rationalsplineform.Changeof illumination

ppp � ucv9Uw x^.ucv�.w xVqq�ucv�

(1) (2) (3) (4)

(5) (6) (7) (8)

(9) (10) (11) (12)

(13) (14) (15) (16)Figure1: A databaseof airplanemodels.

We useda two-stageapproachin informationfusion. Featuresinvariant to affine deformationandperspective projectionwerefirst usedto matchthesilhouetteof thequeryairplanewith thesilhou-ettesof thosein the database.We thenemployedthe illumination invariantscomputedon objects'interior to disambiguateamongmodelswith sim-ilar shapebut differentcolors. The resultsshowthat wewereable to achieve 100% accuracyus-ing our invariants formulation for a databasecomprising very similar models,presentedwithquery imagesof largeperspective shapedistor-tion and changein illumination.

Table2 shows the performanceof usingaffineandperspective invariantsfor shapematchingun-

der a large changeof viewpoint. For eachqueryimage(A throughK), theaffineandperspective in-variantsignatureswerecomputed,and comparedwith the signaturesof all modelsin the database.Correlationcoefficientsasdescribedin Sec.2 wereusedto determinethesimilarity betweeneachpairof signatures.Eachrow in Table2 refersto aqueryimage.Eachof thetencolumnsrepresentstherankgiven to eachairplanemodel from the database(shown in parentheses).The columnsareorderedfrom left to right, with the leftmostcolumnbeingthe bestmatchfound. Only the top ten matchesareshown. Thevalues(not in parentheses)arethecorrelationcoefficients.Entriesprintedin boldfacearetheexpected(correct)matches.

(A) (B) (C) (D)

(E) (F) (G) (H)

(I) (J) (K)Figure2: Thesameairplanesin varyingposesandillumination.

As can be seen,all query imageswere identi-fied correctly. Fig. 3 shows a sampleresult. Theleftmostimageis the queryimage. The top threematchesarein thenext threecolumns—thequeryimage (solid) and estimatedimage (dashed,us-ing perspective invariants)with thecorrespondingdatabasemodelareshown.

For this experiment,all queryimageswerecor-rectlymatchedwith themodelsfrom thedatabase,usingaffine andperspective invariants. However,the error valuesof the top two matchesfor, say,airplaneK werevery closeto eachother. This isbecausethe top two matcheshave similar shapesandbotharesimilar to thequeryimage.Theconfi-dencein theselectedmatchescanbestrengthenedby testingwhetherthe interior regionsof the ob-jects are also consistent. Illumination invariantsreadilyapplieshere.

For illumination invariants, a characteristic

curve wasuniquelydefinedon thesurfaceof eachairplane model in the database(performedoff-line), so that its superimpositionover the modelemphasizesimportant (or interesting)color pat-terns in the image. Our perspective invariantsschemecomputedthe transformationparametersthatbestmatchthetwo givencontours.Thesameparameterswereusedto transformthe character-istic curve definedfor eachmodel to its assumedposein thequeryimage.Hence,thecolorsdefinedby the characteristiccurve in the model shouldmatchthecolorsdefinedby thetransformedcurvein the query image(except for changesdueto il-lumination). Illumination invariantsignaturesforthe query imageswere thencomputed,andcom-paredwith thesignaturesstoredin thedatabaseus-ing Eq.4.

We show one result of illumination invariantswherethe (perspective invariant)errorsof the ¿

Rank(usingaffineandperspectiveinvariants)Image

9Á+Â ÃVÄVÅ ÆVÇUÅ ÈVÂdÉ Ê

100 150 200 250 300 350 400 450 500 550 60050

100

150

200

250

300

350

400

450

100 200 300 400 500 600 70050

100

150

200

250

300

350

400

450

100 150 200 250 300 350 400 450 500 550 6000

50

100

150

200

250

300

350

400

450

( ¿ À @ ) ( X Ð t ) ( ÑcÒ t )Figure3: QueryimageA, with thetop threematchesfrom thedatabase,usingperspective invariants.(Solidfor thequeryimage,dashedfor theestimatedimageusingperspective invariants.)

0 10 20 30 40 50 600

1

2

3

4

5

6

7

8

9

10

(a) (b) (c)

0 10 20 30 40 50 600

1

2

3

4

5

6

7

8

9

10

(d) (e) (f)Figure4: (a),(d) Airplane modelswith the characteristiccurvessuperimposed,(b),(e) query imagewiththe transformedcharacteristiccurvessuperimposed,and(c),(f) illumination invariantsignaturesfor queryimageK (solid)andfor models14and3 (dashed).