FVS: A Content-Based Retrieval Librarylibfvs.sourceforge.net/docs/fvs-manual-1.0.2.pdf · FVS 3.1 Introduction FVS is the name of the library providing the 2-D image and 3-D object

FVS: A Content-Based Retrieval Library

Simon Goodall

June 28, 2006

Contents

1 Acknowledgements 5

2 Introduction 6

3 FVS 73.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Supported file types . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.4 Platform support . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.4.1 MySQL Considerations . . . . . . . . . . . . . . . . . . . 83.4.2 Linux Considerations . . . . . . . . . . . . . . . . . . . . . 83.4.3 Win 32 Considerations . . . . . . . . . . . . . . . . . . . . 8

3.5 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.5.1 Adding new algorithms . . . . . . . . . . . . . . . . . . . 93.5.2 Adding new file formats . . . . . . . . . . . . . . . . . . . 10

3.6 Using FVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.6.1 On the command line . . . . . . . . . . . . . . . . . . . . 103.6.2 MySQL Module . . . . . . . . . . . . . . . . . . . . . . . 11

3.6.2.1 fvs generatefv . . . . . . . . . . . . . . . . . . . 113.6.2.2 fvs comparefv . . . . . . . . . . . . . . . . . . . 123.6.2.3 fvs normalise . . . . . . . . . . . . . . . . . . . . 123.6.2.4 fvs update norm . . . . . . . . . . . . . . . . . . 13

3.6.3 FVS Library . . . . . . . . . . . . . . . . . . . . . . . . . 13

4 The 2-D Algorithms 144.1 The CCV Matching Module . . . . . . . . . . . . . . . . . . . . . 14

4.1.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 144.1.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 144.1.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 144.1.4 Technical Description . . . . . . . . . . . . . . . . . . . . 144.1.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 The Colour Histogram Matching Module . . . . . . . . . . . . . . 174.2.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 174.2.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 17

1

4.2.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 184.2.4 Technical Description . . . . . . . . . . . . . . . . . . . . 184.2.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 19

4.3 The Lab Colour Histogram Matching Module . . . . . . . . . . . 204.3.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 204.3.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 204.3.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 204.3.4 Technical Description . . . . . . . . . . . . . . . . . . . . 20

4.4 The Colour Picker Matching Module . . . . . . . . . . . . . . . . 204.4.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 214.4.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 214.4.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 214.4.4 Technical Description . . . . . . . . . . . . . . . . . . . . 214.4.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 22

4.5 The Mono-Histogram Matching Module . . . . . . . . . . . . . . 244.5.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 244.5.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 244.5.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 244.5.4 Technical Description . . . . . . . . . . . . . . . . . . . . 244.5.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 26

4.6 The Hu Moments Matching Module . . . . . . . . . . . . . . . . 274.6.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 274.6.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 274.6.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 274.6.4 Technical Description . . . . . . . . . . . . . . . . . . . . 27

4.7 The MCCV Matching Module . . . . . . . . . . . . . . . . . . . . 274.7.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 284.7.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 284.7.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 284.7.4 Technical Description . . . . . . . . . . . . . . . . . . . . 284.7.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 29

4.8 The Multi-Scale Mono-Histogram Matching Module . . . . . . . 304.8.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 304.8.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 304.8.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 304.8.4 Technical Description . . . . . . . . . . . . . . . . . . . . 304.8.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 31

4.9 The Pyramid Wavelet Transform Matching Module . . . . . . . . 324.9.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 324.9.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 334.9.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 334.9.4 Technical Description . . . . . . . . . . . . . . . . . . . . 334.9.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 33

4.10 The Query by Fax Matching Module . . . . . . . . . . . . . . . . 354.10.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 354.10.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 35

2

4.10.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 364.10.4 Technical Description . . . . . . . . . . . . . . . . . . . . 364.10.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 37

4.11 The MPWT Matching Module . . . . . . . . . . . . . . . . . . . 374.11.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 374.11.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 384.11.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 384.11.4 Technical Description . . . . . . . . . . . . . . . . . . . . 38

5 The 3-D Algorithms 395.1 The Area Volume Matching Module . . . . . . . . . . . . . . . . 39

5.1.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 395.1.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 395.1.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 395.1.4 Technical Description . . . . . . . . . . . . . . . . . . . . 395.1.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 40

5.2 The Cord Histogram Matching Modules . . . . . . . . . . . . . . 415.2.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 415.2.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 415.2.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 42

5.2.3.1 Cord Hist 1 . . . . . . . . . . . . . . . . . . . . . 425.2.3.2 Cord Hist 2 . . . . . . . . . . . . . . . . . . . . . 425.2.3.3 Cord Hist 3 . . . . . . . . . . . . . . . . . . . . . 425.2.3.4 Cord Histogram . . . . . . . . . . . . . . . . . . 42

5.2.4 Technical Description . . . . . . . . . . . . . . . . . . . . 425.2.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 43

5.3 The ShapeD2 Matching Module . . . . . . . . . . . . . . . . . . . 475.3.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 475.3.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 475.3.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 475.3.4 Technical Description . . . . . . . . . . . . . . . . . . . . 475.3.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 47

5.4 The EGI Matching Module . . . . . . . . . . . . . . . . . . . . . 495.4.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 495.4.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 495.4.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 495.4.4 Technical Description . . . . . . . . . . . . . . . . . . . . 495.4.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 49

5.5 The 3D Hough Matching Module . . . . . . . . . . . . . . . . . . 525.5.1 Module Features . . . . . . . . . . . . . . . . . . . . . . . 525.5.2 Module Properties . . . . . . . . . . . . . . . . . . . . . . 525.5.3 Module Parameters . . . . . . . . . . . . . . . . . . . . . 525.5.4 Technical Description . . . . . . . . . . . . . . . . . . . . 525.5.5 Example Results . . . . . . . . . . . . . . . . . . . . . . . 53

3

6 Distance Metrics and Normalisation 556.1 Minkowski Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.2 Histogram Intersection . . . . . . . . . . . . . . . . . . . . . . . . 556.3 Bhattacharyya Distance . . . . . . . . . . . . . . . . . . . . . . . 556.4 Kullback-Leibler Divergence . . . . . . . . . . . . . . . . . . . . . 556.5 Chi-Squared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566.6 Quadratic Distance . . . . . . . . . . . . . . . . . . . . . . . . . 566.7 Normalisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

4

Chapter 1

Acknowledgements

The images in this document have come from various sources which have beennoted where they are used. Many of the images for the 2-D algorithms comefrom the Victoria and Albert Museum. For the 3-D algorithms, images comefrom both the SCULPTEUR database http://www.sculpteurweb.org (notfully public) and from the Princeton Shape Benchmark http://shape.cs.princeton.edu/benchmark/. Copyright for these images remains with theirrespective owners.

5

Chapter 2

Introduction

This document provides the manual to FVS, the 2-D image and 3-D object pro-cessing library in the developed for the SCULPTEUR project. This documentbegins with a description of the FVS library itself. This is then followed by adescription of each of the 2-D and 3-D algorithms in FVS, including any op-tional parameters they can take. Finally a description of the distance metrics inFVS is presented along with a description of the score normalisation technique.Publications relating to the 3-D work in SCULPTEUR include [9, 8, 3, 2].

6

Chapter 3

FVS

3.1 Introduction

FVS is the name of the library providing the 2-D image and 3-D object pro-cessing functions. FVS is based upon the FVG library produced as part of theARTISTE project [12]. The main differences between FVS and FVG are thatFVS supports 3-D objects and it also provides a MySQL [1] module that al-lows generation and comparison of feature vectors stored directly in a databasetable. FVS provides the ability to generate feature vectors describing featuresof 2-D and 3-D objects, comparison between feature vectors, and a thumbnailgenerator to provide a small view of a query object.

FVS is a library with a command line interface and MySQL module toprovide the interfaces.

3.2 Supported file types

Technically, FVS can read 2-D images in any format supported by ImageMagick,an image processing library used by VIPS [6] to load images. In the distributedbinaries however, only JPEG [10] and TIFF [18] file formats are supported due tothe complexity of compiling the necessary dependencies on all target platforms.

FVS supports three 3-D file formats. These are VRML 97 [5] (both com-pressed and uncompressed), OFF, and TRI. It must be noted here that VRML1.0 is not a supported 3-D file type.

3.3 Dependencies

FVS has two main dependencies; these are VIPS for 2-D image loading andsome processing operations, and Cyber X3D [11] for parsing VRML 97 objects.These two libraries however have their own dependencies.

7

Due to more recent additions to the VIPS library that add a pthread depen-dency in VIPS 7.10.x, VIPS 7.8.x is the best version to use with the MySQLmodule.

3.4 Platform support

FVS is supported on both Linux platforms and Win 32 platforms. It has alsobeen run on Mac OS-X. Providing the dependencies can be met, FVS shouldwork on additional platforms. The feature vector I/O code ensures that datais stored in an endian independent manner which should allow FVS to run on“Big Endian” architectures.

3.4.1 MySQL Considerations

MySQL can dynamically load modules to add functionality. The new function-ality can be accessed through a normal SQL based query. There are severalcaveats to producing a module. The first is that the module should be compiledstatically, that is so that all the module code, and that of its dependencies arecompiled into a single file. It might be possible to get a module to dynamicallyload its dependency libraries, however this would require the correct versionsto be available on the system, and have MySQL configured so that it can findthem. Initial experiments in distributing FVS like this proved problematic.

The second caveat is that as MySQL is a C based application, certain C++features cannot be used. The main feature is C++ exceptions, which causeMySQL to crash as soon as an exception is thrown. The FVG library usedthe C++ version of VIPS which used exceptions for error handling. FVS wasre-written to use the C version of VIPS to solve this problem.

The third caveat is that the module needs to be thread safe, and cannotbe compiled with pthreads. This is especially important as VIPS makes useof threads to improve its performance, and several of its dependency librariesmake use of pthreads.

3.4.2 Linux Considerations

The last couple of years have seen many advances in the GNU C++ com-piler [16], gcc. Unfortunately, these advances have broken compatibility betweenthe various versions of the compiler. Typically this means that programs com-piled on the more recent versions of gcc will not work on systems hosting olderversions of gcc. However programs compiled on the older gcc versions will workon systems hosting the newer versions.

3.4.3 Win 32 Considerations

As most of the software is written in Linux, FVS is cross-compiled into a Win32 DLL. This requires that all dependencies can be cross-compiled. Not every

8

library is in a format that easily allows this (or is able to be cross-compiled),and as such has led to the small number of 2-D image formats supported byFVS.

3.5 Architecture

FVS is written as a library and has two front-ends to provide access to itsfunctionality. The main front-end is the MySQL module used in the core of theSCULPTEUR system. The other front-end is a command line tool.

3.5.1 Adding new algorithms

Adding a new algorithm is fairly straight forward. However there are severalfiles that need to be edited to first compile the new algorithm code, and secondlytell fvs that it is available.

FVS uses the autoconf [14] and automake [15] for its build system. Addinga new algorithm requires editing the existing configure.in and Makefile.am files,and creating a new one for the algorithm.

The first stage to creating a new algorithm is to take an existing algortihm(such as CCV for a new 2D algorithm, or ShapeD2 for a new 3D algorithm) andcopy the all the code files and the Makefile.am into the new algorithm directory.

Typically the structure is as follows;

fvs-VERSION/fvs/AlgName/Makefile.amfvs-VERSION/fvs/AlgName/FA_AlgName.cppfvs-VERSION/fvs/AlgName/FA_AlgName.hfvs-VERSION/fvs/AlgName/FV_AlgName.cppfvs-VERSION/fvs/AlgName/FV_AlgName.h

Code files prefixed with FV denotes the code files for the class that handlesthe storeage of the feature vector and its reading/writing functions. Code filesprefixed with FA denotes the code files for the class that handles the generationand comparison of the feature vectors. These files need to exist and have theappropriate defininations. Additional code files can be used if required.

Once copied and appropriately renamed, Makefile.am needs to be updatedfor the new file names, and the class definitions need to be updated. The codeshould also be changed to use the algorithm code.

Then fvs-VERSION/fvs/Makefile.am needs to have the new algorithm direc-tory added to the list of subdirs, specified on the line beginning with“SUBDIRS= ColourHistogram...”. The .la file created for the algorithm also needs to beadded to the libfvs 1 0 la LIBADD declaration.

Finally, configure.in needs to add the new Makefile to its list of generatedfiles. Add fvs/AlgName/Makefile to the end of the list in the AC OUTPUTblock.

This completes the addition of the algorithm to the build system. To tellFVS about the new algorithm, the header files need to be added to the list

9

of includes in fvs/fv.h and a REGISTER FV(AlgName) needs to be added atthe bottom of the file. Finally, fvs/ifipids.h defines the ID and name for eachalgorithm. Add two #define statement in the form #define AlgName Name“AlgName” and #define AlgName ID XXXX where AlgName is the name ofthe algorithm (no spaces and the first character must be a letter) and XXXX isa unique ID number.

Once completed, the new algorithm should now be integrated into FVS. Theexisting algorithm implementations should be used as examples on working withthe internal functions.

3.5.2 Adding new file formats

Adding new 2-D file formats to FVS requires making the appropriate changesto Vips and is not an issue for FVS. Adding new 3-D file formats however is adirect issue for FVS.

FVS has an internal 3-D representation that each of the file format loadersneeds to be able to produce. This format stores all of the index, vertex, normaland texture information. Any transformations need to be applied to the databefore passing to FVS as they cannot be stored directly in the representation.

3.6 Using FVS

FVS can be used on the command line, as a library, or as a MySQL module.

3.6.1 On the command line

FVS has two main modes of functionality. These are generate and compare.To generate a feature vector, FVS can be invoked with the following com-

mand;

fvs -f <algorithm name> [-m] -g <input file name>[-o <output file name>][--ascii]

where <algorithm name> is to be substitued with the algorithm name or ID,<input file name> is the path to the input 2-D image or 3-D object. Optionally,<output file name> specified the filename of the resulting feature vector. Bydefault this will be <input file name>.<algorithm name>[.ascii].ifo . The –ascii option specifies to create an ASCII based feature vector, rather than abinary version. This is denoted by the [.ascii] part of the default feature vectorname. [-m] is a flag to enable the multiscale wrapper for 2-D algorithms.

To compare two feature vectors, FVS can be invoked as follows;

fvs -f <algorithm name> [-m] -c <query file name><reference file name>

10

where <algorithm name> is to be substitued with the algorithm name or ID,<query file name> is the path to the query feature vector and <reference filename> is the path to the reference feature vector. [-m] is a flag to enable themultiscale wrapper for 2-D algorithms.

Additionally, both these commands can take a string of options to pass tothe algorithm. These can either be integer values, real values or string values.Appending –integer i will pass an integer with the value i to the algorithm.Similarly use –real r for real numbers and –string str for string values. See thealgorithm descriptions for details on what parameters to pass to the algorithms.Please note that the ordering of parameters is important.

Finally, normalisation can be used by using one of several additional argu-ments.

To initialise a normalisation blob, call fvs as follows;

fvs --init-norm <norm file name>

where <norm file name> is the file name for the resulting normalisation blob.By appending –integer i and –real r the number of bins and the maximum scorevalue can be specified respectively.

To update a normalisation blob, perform a comparison as normal, and usethe –update-norm argument as follows;

fvs -f <algorithm name> [-m] -c <query file name><reference file name> --update-norm <norm file name>

where <norm file name> is the file name of the normalisation blob.To normalise a score, perform a comparison as normal and use the –normalise-

score argument as follows;

fvs -f <algorithm name> [-m] -c <query file name><reference file name> --normalise-score <norm file name>

where <norm file name> is the file name of the normalisation blob.

3.6.2 MySQL Module

The module has four functions. Below is the standard interface to the functions.Additionally both the fvs generatefv and fvs comparefv functions can take a listof optional arguments specified after the required options. See the details of eachalgorithm for what options are available.

3.6.2.1 fvs generatefv

This function generates a feature vector of the specified type for the object inthe given filename. If the multiscale wapper flag is set to 1, then the wrappermodule is used.

Returns STRING: The generated feature vector blob, or NULL on error.

11

Funtion parameters:Type What DescriptionSTRING / IN-TEGER

Feature VectorName / ID

This is the name or ID of the featurevector type.

INTEGER Multiscalewrapper flag

The flag indicates whether to use themultiscale wrapper. (0 for false, 1 fortrue).

STRING Input file name This is the filename of the file to gener-ate the feature vector from.

3.6.2.2 fvs comparefv

This function takes two feature vector blobs and returns a dissimilarity score.Returns REAL: The dissimilarity score. If score is less than 0, an error

occurred.Error Code Reason

-1.0 General Error during comparison.-2.0 One or both fv’s are NULL pointers-5.0 Failed to create FeatureAPI object-10.0 Error loading memoryblob into fv object-11.0 Error loading memoryblob into fv-12.0 Feature vector has zero length

Function parameters:Type What DescriptionSTRING / IN-TEGER

Feature VectorName / ID

This is the name or ID of the featurevector type.

INTEGER Multiscalewrapper flag

The flag indicates whether to use themultiscale wrapper. (0 for false, 1 fortrue).

STRING Query FV Blob The query feature vector blob.STRING Reference FV

BlobThe reference feature vector blob.

3.6.2.3 fvs normalise

This function normalises a score according to the specified normalisation blob.Returns REAL: The normalised score.Error Code Reason

-1.0 Normalisation Blob is NULL-2.0 Normalisation Blob has no length-3.0 There was an error parsing the blob

Function Parameters:Type What Description

STRING Normalisation Blob This is the blob containing the normalisation dataREAL Score The score to normalise

12

3.6.2.4 fvs update norm

This function adds a score to the normalisation blob. If the input blob is NULL,then a new blob is created.

Returns STRING: The updated blob. Returns NULL on error.Type What Description

STRING Normalisation Blob This is the blob containing the normalisation dataREAL Score The score to add to the blob.

3.6.3 FVS Library

All of the functionality of FVS is contained within the FVS library. The FVScommand line and FVS MySQL Module are front-ends to this library. Pleaserefer to these for examples in using the FVS library.

13

Chapter 4

The 2-D Algorithms

4.1 The CCV Matching Module

Algorithm By: Stephen ChanName: CCVID: 1000

4.1.1 Module Features

The CCV module [13] allows retrieval of similar images based on the generalcolour distribution of the image, with discrimination between colours in imageswhich are homogeneous to some sizable area.

The module is good for retrieval of images based on a known query imagewhere the retrieved images are required to have a similar mix of colours. It isnot suitable for retrieval of images based on a query image which is only partof a complete image.

4.1.2 Module Properties

Module Speed: FastModule Accuracy: Medium

4.1.3 Module Parameters

Generate: This module has no generation parameters.Compare: This module has no comparison parameters.

4.1.4 Technical Description

The CCV stands for Colour Coherence Vector. A coherent region of colours inan image is a region of colour which is larger than some threshold. This moduleretrieves images which have similar distributions of coherent colours.

14

A histogram of 64 bins (4x4x4) is generated for both coherent and incoherentcolours and these are matched separately. As for Colour-Histogram, 64 binshas been chosen as a trade-off between accuracy and speed. The total size ofthe feature vector for a CCV is therefore 64x2 integer numbers (512 bytes perfeature).

Coherence and incoherence are arbitrarily defined as greater and less then5% of the total image area, respectively. This means if a pixel is part of a regionwhich is less than 5% of the total image area it is added to the incoherenthistogram within the CCV. If it is greater than 5% of the total image area it isadded to the coherent histogram within the CCV.

Here is an example:

The chessboard image is 50% black and 50% white, arranged into 64 squares,where 32 are white and 32 are black. Each square constitutes a region each ofwhich is 1/64th of the total image area. This is approximately 1.5% of the imagearea. They are therefore considered incoherent and stored into the incoherenthistogram, leaving the coherent histogram empty. The second image containsthe same amount of black and white (50% of each), however, the two regionsare coherent and are therefore stored into the coherent histogram, leaving theincoherent histogram empty.

When each coherent and incoherent histogram is matched against each other,these two images will have a large distance, as they should indeed do, unlike astandard histogram function which would give a distance of zero implying theywere identical.

4.1.5 Example Results

The following is an example query giving what would be considered good results.Note: The matching is based only upon the general colour distribution of the

image, with discrimination between colours in images which are homogeneousto some sizable area.

15

Query 1 2 3

4 5 6Images courtesy of Victoria and Albert Museum

In the above example, the query image is contained within the database. Itis therefore expected to be found in first place, as it is. This is a good test ofthe integrity of the algorithm. Also, all the retrieved results contain areas ofcontiguous colour similar to that of the query image. Note that the contiguouswhite background will alter the match, because it would become a coherentregion within the vector.

The CCV is a whole-image query algorithm, and cannot be used to findsub-images, because it is likely that the sub-image will have a different colourdistribution. An image which is a sub-image of a database image may be usedas the query but the parent from which it was derived is unlikely to be found.The query below shows an example of using an image which is a sub-image of animage in the database as the query. Notice the results all have a similar colourdistribution (white/yellow), but it did not find the image from which the queryimage was taken.

16

Query 1 2 3


Used on its own, you should definitely not expect the CCV algorithm to beable to find specific instances of objects (e.g. chairs, pots, etc). However, usedwith a metadata search to locate similar types of object, this algorithm couldlocate those of a similar colour.

4.2 The Colour Histogram Matching Module

Algorithm By: David DupplawName: ColourHistogramID: 1024


The Colour Histogram module [17] allows retrieval of similar images based onthe general colour distribution of the image.

The module is good for retrieval of images based on a known query image,if the retrieved images are required to have a similar mix of colours. It is notsuitable for retrieval of images based on a query image which is only part of acomplete image.


Module Speed: FastModule Accuracy: Low

17


Generate: This module takes 3 positive integer parameters specifying thenumber of bits for the reg, green and blue colour channel re-spectively.

Compare: This module has no comparison parameters.


This histogram matching algorithm simply uses the frequency of occurrence ofeach colour of the histogram within the image. The more of a particular colouran image contains the higher its frequency will be within the histogram.

The histogram is made with 64 bins, a compromise between speed (it takeslonger to match more bins), and accuracy (the less bins the less discriminat-ing the results would be - i.e. images which are less similar would have lowerdistances).

Before matching histogram are normalised so that the area underneath thehistogram is unity (normalised by the number of pixels in the image). Thismeans that the size of the image will not affect the match (other than anyexpected aliasing).

Here is an example:

Vase Images courtesy of Victoria and Albert Museum

Note in the second example, how the background is dominant in the image.Because the colours of the pots are spread fairly evenly over the bins, and thebackground is pre-dominantly one colour, and therefore one bin of the histogram,once the histogram is normalised the background dominates the vector. This is

18

the main drawback of histogram matching: the background information is allincluded within the feature and it cannot be ignored. In the first example, thebackground is more evenly distributed over bins (due to the shading) and theobject groups into a few bins (due to its flat colour distribution). Therefore thebackground does not dominate the histogram so much.


The following is an example query giving what would be considered good results.Note: The matching is based only upon the general colour of the images.

Query 1 2 3


Because the histogram is such a basic algorithm, there are no cases whereit should give unexpected results. However, notice that the background (whichis effectively ”noise” in this context) affects the match, so that the results allcontain large amounts of white, and some green and browns.

Used on its own you should definitely not expect the ColourHistogram al-gorithm to be able to find specific instances of objects (e.g. chairs, pots, etc).However, when used with a metadata search to locate similar types of object,this algorithm could locate those of a similar colour.

19

4.3 The Lab Colour Histogram Matching Mod-ule

Algorithm By: David DupplawName: LabColourHistogramID: 1026


The module allows retrieval of similar images based on the general LAB colourdistribution of the image.

The module is good for retrieval of images based on a known query image,if the retrieved images are required to have a similar mix of colours. It is notsuitable for retrieval of images based on a query image which is only part of acomplete image.




Generate: This module takes three integer values specifying the number ofbits for the L, A and B colour channels respectively.



This histogram matching algorithm simply uses the frequency of occurrence ofeach LAB colour of the histogram within the image. The more of a particularcolour an image contains the higher its frequency will be within the histogram.


Before matching histogram are normalised so that the area underneath thehistogram is unity (normalised by the number of pixels in the image). Thismeans that the size of the image will not affect the match (other than anyexpected aliasing).

4.4 The Colour Picker Matching Module

Algorithm By: David DupplawName: ColourMatchID: 1023

20


The module allows retrieval of images based on the images containing certainamounts of colours. The colours need not be sizable regions, or be within thesame context within images (a property which histograms impose). The re-sults are ordered based on how similar the amounts of colours in the image arecompared to the colour selection.

The module is good for retrieval of images based on a single or multiplecolour queries.


Module Speed: Very FastModule Accuracy: High


Generate: Not applicable - No generate functionCompare: This module takes a real value setting the fuzziness of the com-

parison. It can also take three integer values specifying an RGBcolour. If both sets of parameters are specified, then fuzzinessmust be first.


The colour picker uses histogram features generated by the ColourHistogramor LabColourHistogram module. It matches single or multiple colours in thehistogram in a similar way to the colour histogram matching code, except thatit matches only those colours in the query histogram.

A histogram (of any size) is generated for each object in the database. Cur-rently we use a 64 bin RGB colour histogram (4x4x4) or a 216 bin Lab colourhistogram (6x6x6).

When the colour picker is invoked, an appropriate histogram is generatedfrom the selected colours. The values in the histogram represent the valuesselected.

Figure 4.1: Red colour selection, gives a histogram with a single red bin popu-lated

21

Figure 4.2: A two colour selection gives a histogram with two bins populated

Figure 4.3: The amounts of selected colour are reflected in the histogram

The matching matches each populated bin in the query histogram with thematching bin in the match histogram. Because the idea is to find images con-taining these colours, it is pragmatic to disregard, or give a large score, to thoseimages which do not contain those colours. This requires a predicate for colourinclusion. The predicate is based on a percentage value of the given amount ofcolour (a fuzziness rating). For example, if the fuzziness rating was 50%, a binin the match histogram would be considered a result if the value fell within +/-50% of the value of the query histogram’s bin. By default the rating is near to100%, so that if a bin contains any amount of the given colour it will be consid-ered a match, although results will be ordered based on how similar the amountsare to the query. The fuzziness should not be 100% else a zero amount of thecolour will match. If the match histogram’s bin value falls outside this range(or is zero) then the match is given a maximum score (disregarded as a matchbecause it does not contain all the colours specified) and no more matchingtakes place for this feature.


The following is an example query giving what would be considered good results.Note: The matching is based upon the inclusion of the selected colour.

22

Query 1 2 3


Query 1 2 3


Note that many of the images contain colour charts, and this could affectthe match for obvious reasons (see result 6 of first match, and result 5 of second

23

match).

4.5 The Mono-Histogram Matching Module

Algorithm By: Stephen Chan and David DupplawName: HistogramMonoID: 1015


The module allows retrieval of similar images based on the general distributionof brightnes in the image. It is really only suitable for monochromatic images,such as monochrome photography or radiography.

A monochrome histogram contains the frequency (i.e. number of pixels) ofeach brightness level (of which there are 64 in this module) that occurs withinthe image.

The module is good for retrieval of images based on a known query image ifthe retrieved images are required to have a similar overall brightness distribu-tion. It is not suitable for retrieval of images based on a query image which isonly part of a complete image.




Generate: This module takes a single integer specifying the number of his-togram bins.



This monochrome histogram matching algorithm simply uses the frequencyof occurrence of each level of brightness of the histogram within the image.The more of a particular brightness an image contains the higher its frequencywill be within the histogram. Colour images in the database are converted tomonochrome for matching with this algorithm, by converting RGB values tomonochrome with (R+G+B)/3.


Before matching histograms are normalised so that the area underneath thehistogram is unity (normalised by the number of pixels in the image). This

24

means that the size of the image will not affect the match (other than anyexpected aliasing).

Here is an example:

Images courtesy of Victoria and Albert Museum

As with the ColourHistogram the background has an effect on the histogram,and therefore on the matching. In the second example the dark background isincreasing the frequencies of the dark bins in the histogram.

The reason a monochromatic histogram is required, is that the colour his-togram is not discriminating enough for monochromatic images. A 64 bin colourhistogram has only 4 bins dedicated to grey-scale values. This means that mostgrey-scale images would look similar in a colour histogram. The histogramsbelow show an image in both RGB and monochrome histogram space.

25

Images courtesy of Victoria and Albert Museum


The following is an example query giving what would be considered good results.Note: The matching is based only upon the general distribution of brightness

in the image.

Query 1 2 3


The monochrome histogram is specifically designed for matching with monochromeimages. It will also work with colour images, however. The necessity for havingboth colour and monochrome histogram is that the colour histogram, being de-

26

signed for colour, is very inaccurate for grey-level images. The monochrome his-togram algorithm is around 13 times more discriminating between monochromeimages than the colour histogram is between monochrome images, because itcontains 13 times more grey-levels.

The above example shows a large monochromatic image being used as aquery within a small 1000 set database containing both black and white andcolour images. Because the scope of the query was not limited to monochromeimages, the retrieval found both colour and monochrome images. Notice thatthe monochrome images it found belonged to the same painting. This will notnecessarily happen in all datasets, as it depends on the deviation of the queryfrom the other images.

Used on its own you should definitely not expect the monochrome histogramalgorithm to be able to find specific instances

4.6 The Hu Moments Matching Module

Algorithm By: M Faizal A FauziName: HuMomentsID: 1070


The Hu-Moments are a set of seven rotation, scale and translation invariantmoments.


Module Speed: FastModule Accuracy: <accuracy>




This module first converts an image into a black and white representation settingthe threshold at the average image intensity. The moments are then applied tothe converted image.

4.7 The MCCV Matching Module

Algorithm By: Stephen ChanName: MCCVID: 1002

27


The MCCV module [4] allows retrieval of similar images based on the generalcolour distribution of the image and sub-images, with discrimination betweencolours in images which are homogeneous to some sizable area.

The module is good for retrieval of images based on a known query imageif the retrieved images are required to contain an area with a similar mix ofcolours to the query image. This means it is also suitable for retrieval of imagesbased on a query image which is only part of a complete image.

This module divides database images into a hierarchy of small patches andthe CCV is calculated for each patch. The query is compared with all patchesso that sub-images may be located in the parent image.


Module Speed: SlowModule Accuracy: Very-High




The detail finder finds sub-images by dividing the query and the database imageinto a number of tiles over a number of resolutions calculating the CCV for eachof the tiles.

Figure 4.4: Multiscale Pyramid Structure

The highest resolution image is converted into 64x64 tiles, overlapping by32 pixels in each dimension. The image resolution is halved and, again, dividedinto 64x64 tiles (of which there will be 4 times less). The lowest resolution is of64x64 pixels and 1 single tile.

28

For each tile a CCV is created and stored, so that the final feature vector isa set of CCV feature vectors, one for each tile.

Both the query image and the database image are converted into a pyramidstructure, and then each of the tiles in the query image are compared againsteach of the features for the tiles in the database image using the CCV matchingalgorithm, as described on the CCV Help Page.

The query is converted to a pyramid to facilitate the database image beinga subimage of the query image (double sub-image detection). An alternative isto assume the query is a subimage of the database image only, and perform aCCV match of the whole query image against each of the tiles in the databaseimage.


The following is an example query giving what would be considered good results.Note: The matching is based only upon the general colour layout of the

images and subimages.

Query 1 2 3


In the above example, all the retrieved results contain areas of contiguousskin like colour, and areas of contiguous dark-brown in a similar way to thequery. Also note, that the query image, despite its diminutive size, has beenfound in the original image from which is was taken.

Because the MCCV is a multi-scale version of the CCV, you can expect allthe same types of properties and shortcomings. Used on its own you shoulddefinitely not expect the MCCV algorithm to be able to find specific instances

29

of objects (e.g. chairs, pots, etc). However, when used with a metadata searchto locate similar objects, this algorithm could locate those of a similar colour.

4.8 The Multi-Scale Mono-Histogram MatchingModule

Algorithm By: Stephen Chan and David DupplawName: MHistogramMonoID: 1017


The module allows retrieval of similar images based on the general distributionof brightness in the image and sub-images. It is only suitable for monochromaticimages, such as black and white scans from IR-reflectance, or similar. Colourimages are converted to monochrome before comparison using this module.

This module divides database images into a hierarchy of small patches andthe monochrome histogram is calculated for each patch. The query is comparedwith all patches so that sub-images may be located in the parent image.

The module is good for retrieval of images based on a known query image ifthe retrieved images are required to contain an area with a similar distributionof brightness to the query image. This means it is also suitable for retrieval ofimages based on a query image which is only part of a complete image.


Module Speed: SlowModule Accuracy: High




The detail finder finds sub-images by dividing the query and the database im-age into a number of tiles over a number of resolutions calculating the Monohistogram for each of the tiles.

30



For each tile a Mono histogram is created and stored, so that the final featurevector is a set of Mono histogram feature vectors, one for each tile.

Both the query image and the database image are converted into a pyramidstructure, and then each of the tiles in the query image are compared againsteach of the features for the tiles in the database image using the Mono histogrammatching algorithm, as described on the Mono histogram Help Page.

The query is converted to a pyramid to facilitate the database image beinga subimage of the query image (double sub-image detection). An alternative isto assume the query is a subimage of the database image only, and perform aMono histogram match of the whole query image against each of the tiles in thedatabase image.


The following is an example query which would be considered good results.Note: The matching is based only upon the general brightness levels of the

images and subimages.

31

Query 1 2 3


The multi-scale monochrome histogram has the advantage over the multi-scale colour histogram in that the results can appear in both grey-level andcolour images. The multi-scale colour histogram will only work with colourimages. The results above show this, as the correct sub-image and sub-imagelocation is found in the monochrome image, and also in the colour image.

Because the multi-scale monochrome histogram is a multi-scale version of theregular monochrome histogram, you can expect all the same types of propertiesand shortcomings. Used on its own you should definitely not expect the multi-scale monochrome histogram algorithm to be able to find specific instances ofobjects (e.g. chairs, pots, etc). However, when used with a metadata searchto find similar objects, this algorithm could locate those of a similar brightnessdistribution.

4.9 The Pyramid Wavelet Transform MatchingModule

Algorithm By: M Faizal A FauziName: PWTretrieval

ID: 1019


The PWT module [7] allows retrieval of similar images based on the generaltexture distribution of the image where image texture here, is concerned with

32

the repeating patterns throughout the whole image.The module is good for retrieval of images based on a known query image

if the retrieved images are required to have a similar texture. It is not suitablefor retrieval of images based on a query image which is only part of a completeimage.


Module Speed: Very FastModule Accuracy: Medium


Generate: This function has no generation parameters.Compare: This function has no comparison parameters.


The PWT decomposes an image based on a wavelet transform, which can bethought of as similar to a Fourier transform, which transforms the iamge domaininto a frequency domain. The frequency components of the image are analysedand a number of descriptors generated which represent the amounts of a discretenumber of frequencies in the image.

(a) Decomposition in image domain (b) Decomposition in frequency do-main

Images are resized to 512x512 to perform this decomposition, which yields22 frequency descriptors for an image. This makes the matching very fast. Thecomparison is achieved using standard Euclidean distance measure.


The following is an example query giving what would be considered good results.

33

Note: The matching is based only on the texture, i.e. repeating patterns, inthe whole image.

Query 1 2 3


The above results show the type of results you could expect to get fromthe PWT. The dataset contains a set of fabrics, and the similarity between therepeating pattern of the query fabric shows up in the results.

You should not expect the PWT to be able to find small amounts of texture,in an image, because it is not multi-scale. So a pot with the surface of a certaintexture, may not find other pots in a dataset with the same surface due tothe other information (such as backgrounds) in the image causing errors in thematching.

34

Query 1 2 3


Used on its own you should definitely not expect the PWT algorithm to beable to find specific instances of objects (e.g. chairs, pots, etc). However, whenused with a metadata search to locate similar types of object, this algorithmcould locate those of a similar texture.

4.10 The Query by Fax Matching Module

Algorithm By: M Faizal A FauziName: QBFID: 1031


This module attempts retrieval when the query image is a low quality monochromeimage, for example a facsimile of a painting which is in the database.

The module is good for retrieval of images based on a known query imageif the retrieved images are required to have a similar layout of dark and lightpixels. It is not suitable for retrieval of images based on a query image which isonly part of a complete image.


Module Speed: Very FastModule Accuracy: Low

35


Generate: This module takes an integer value specifying the type of featureto generate. A value of 1 specifies to generate a feature vectorto be used as a reference. A value of 2 specifies to generate afeature vector to be used as a query (this is the default mode ofoperation).



The query by fax is based upon a set of PWT measures of the image at variousthreshold levels of a monochrome instance of the image. A QbF feature vectorconsists of 99 PWT features at various levels of threshold (between 1% blackand 99% black) of the image.

Figure 4.6: Query By Fax: Database Images converted to 99 PWT levels onleft, query image on right.

Matching is performed by a simple step process:

• Make query fax image binary (it almost is anyway, but this enforces it)

• Detect what percentage of the image is black/white.

• Select from the set of PWT vectors for a database image the vector whichis of the same percentage black/white.

36

• Perform a PWT match, as described on the PWT help page.


The following is an example query giving what would be considered good results.Note: The matching is based upon the layout of dark and light pixels.

Query 1 2 3


4.11 The MPWT Matching Module

Algorithm By: Simon GoodallName: MPWTID: 1033


The module allows retrieval of similar images based on the general texture dis-tribution of the image and sub-images, where image texture here, is concernedwith the repeating patterns throughout the image.

The module is good for retrieval of images based on a known query image ifthe retrieved images are required to have a similar texture.

This module divides database images into a hierarchy of small patches andthe PWT is calculated for each patch. The query is compared with all patchesso that sub-images may be located in the parent image.

37


Module Speed: SlowModule Accuracy: <accuracy>




The detail finder finds sub-images by dividing the query and the database imageinto a number of tiles over a number of resolutions calculating the PWT for eachof the tiles.



For each tile a PWT is created and stored, so that the final feature vectoris a set of PWT feature vectors, one for each tile.

Both the query image and the database image are converted into a pyramidstructure, and then each of the tiles in the query image are compared againsteach of the features for the tiles in the database image using the PWT matchingalgorithm, as described on the PWT Help Page.

The query is converted to a pyramid to facilitate the database image beinga subimage of the query image (double sub-image detection). An alternative isto assume the query is a subimage of the database image only, and perform aPWT match of the whole query image against each of the tiles in the databaseimage.

38

Chapter 5

The 3-D Algorithms

5.1 The Area Volume Matching Module

Algorithm By: Tony TungName: AreaVolumeID: 1080


The AreaVolume module finds objects with a similar area to volume ratio. Thismodule only gives a very simple comparison between objects and should onlybe used as a rough estimator of similarity.




Generate: This module has no generation parameters.Compare: This module takes an integer value determining which compo-

nent to compare. A value of 0 compares surface area, a valueof 1 compares the volume, value of 2 compares the ratio Area

V olume

, and a value of 3 (default) compares the ratio 36 · π.V olume2

Area3 .This module also takes a string parameter specifying which dis-tance metric to use (default is “cityblock”) and a real value asan optional argument to the distance metric.


Area is obtained by summing the area of each triangle of the mesh describingthe surface of the object.

39

Volume is the sum of the tetrahedral formed by the oriented triangles of themesh and the centre of mass of the object.

Volume and area are fast to compute but strongly depend on the meshquality: the 3D model needs to be closed and manifold. We proposed thefollowing formula for the volume-area calculation:

Ratio = 36 · π · volume2

area3

This ratio has no dimension and Ratio=1 for the sphere (surface is minimal).


Here is an example query:

Query 1 2 3

4 5 6Images from the Princeton Shape Benchmark

Figure 5.1: Area Volume Ratio (Princeton Dataset)

As can be seen, the area volume ratio finds the query object as the bestmatch, however the other results are quite dissimilar in the princeton dataset(Figure 5.1).

40

Models are from the SCULPTEUR dataset

Figure 5.2: Area Volume Ratio (SCULPTEUR Dataset)

The Area Volume ratio performs much better on the SCULPTEUR datasetthan the Princeton dataset (Figure 5.2).

5.2 The Cord Histogram Matching Modules

Algorithm By: Tony TungName: CordHist1, CordHist2, CordHist3, CordHistogramID: 1072, 1076, 1078, 1082


The Cord Histogram modules allow retrieval of similar shaped 3D objects basedupon various attribute distributions of Cords. These attributes are Cord length,and the angles between the first and second principal axis.

These modules are good for non-complex objects. They are sensitive to meshconnectivity changes.


Module Speed: FastModule Accuracy: Low-Medium

41


5.2.3.1 Cord Hist 1

Generate: This module takes an integer parameter specifying the numberof histogram bins (default is 16).

Compare: This module takes a string parameter specifying which distancemetric to use (default is “euclidean”) and a real value as anoptional argument to the distance metric.

5.2.3.2 Cord Hist 2



5.2.3.3 Cord Hist 3



5.2.3.4 Cord Histogram

Generate: This module takes three integer parameters specifying the num-ber of histogram bins for CordHist1, CordHist2 and CordHist3respectively (defaults to 16 for each).



Feature vectors are histograms of the cords calculated over the entire meshtriangles.

A Cord is defined as the vector between the centre of mass of an object tothe centre of mass of a triangle on its surface. The Cord Histogram 1 algorithmmakes a histogram of the lengths of these cord for every triangle. Cord His-togram 2 makes a histogram based on the angle between a cord and the firstprincipal axis for every triangle. Cord Histogram 3, does likewise, but againstthe second principal axis. The principal axis are calculated by using PrincipalComponent Analysis. This process is also used to help determine a commonpose for comparison to other objects.

This technique is not discriminating for complex object, and not invariantto mesh connectivity changes.

42


Here are some example results for each of the Cord Histogram variants.

Query 1 2 3


Figure 5.3: Cord Histogram 1 (Princeton Dataset)

For the Cord Hist 1, we can see that it can find several other similar objects,but it also finds the snake object as the second best match which would not beexpected (Figure 5.3).


Figure 5.4: Cord Histogram 1 (SCULPTEUR Dataset)

The Cord Hist 1 does reasonable well on the SCULPTEUR dataset, howeverthere are a couple of unexpected results (Figure 5.4).

43

Query 1 2 3



The Cord Hist 2 does slightly better with all the objects being similarlyshaped, if not all from the same class of objects (Figure 5.5).



With the SCULPTEUR dataset, the Cord Hist 2 retrieves a full set of fairlysimilar objects (Figure 5.6).

44

Query 1 2 3



The Cord Hist 3 algorithm produces similar results. The human model issimilar to a plane model due to the arms sticking out like wings. The vase objectis not expected in these results.



The Cord Hist 3 does not do as well as the Cord Hist 2 on the SCULPTEURdataset (Figure 5.8).

45

Query 1 2 3


Figure 5.9: Combined Cord Histogram (Princeton Dataset)

The combined cord histogram gives the best results out of the different vari-ants. There are a few human models and a fish model, however they are similarin their general shape to that of the query object (Figure 5.9).


Figure 5.10: Combined Cord Histogram (SCULPTEUR Dataset)

Against the SCULPTEUR dataset, the combined cord histogram managesto achieve a full set of vases (Figure 5.10).

46

5.3 The ShapeD2 Matching Module

Algorithm By: Tony TungName: ShapeD2ID: 1074


This module allows retrieval of similar shaped 3D objects based upon the dis-tribution of the distance between random points on the surface of the object.

This module may be used for queries on non-complex objects.




Generate: This module takes an integer parameter specifying the numberof samples to use (default is 10242) and a second integer param-eter specifying the number histogram bins (default is 64).

Compare: This module also takes a string parameter specifying which dis-tance metric to use (default is “euclidean”) and a real value asan optional argument to the distance metric.


The Shape Distribution D2 approach proposed by Princeton University is aprobabilistic method. The principal advantages are the robustness to noise onthe mesh and the geometric invariance properties of the descriptor.

Feature vectors are histogram of distance between points randomly chosenover the surface of the object with a uniform surface repartition.

This method is fast to compute and permits one to describe the main shapeof a 3D object but not the details. It may be used for queries of non complexshape matching and primitive shapes (sphere, torus, ...).


Here is an example query.

47

Query 1 2 3


Figure 5.11: Shape D2 (Princeton Dataset)

As can be seen this algorithm retrieves similarly shaped objects in the Prince-ton Dataset. The two tree objects seem a little out of place at first, howevercomparing them to result 6, the similarity can be seen.


Figure 5.12: Shape D2 (SCULPTEUR Dataset)

The ShapeD2 does not seem to perform as well on the SCULPTEUR dataset,and the only really similar object is in fourth place.

48

5.4 The EGI Matching Module

Algorithm By: Tony TungName: EGIOct, EGISphereID: 1088, 1086


This module allows retrieval of similar 3D objects based upon the Complex EGI(Extended Gaussian Images) method.

This module may used for queries on complex objects.




Generate: This module has no generation parameters.Compare: This module also takes a string parameter specifying which dis-

tance metric to use (default is “euclidean”) and a real value asan optional argument to the distance metric.


The idea of the Extended Gaussian Image (EGI) is to map the 3D mesh trianglesorientation and/or area on a Gaussian sphere.

This is a well-known technique which permits one to deduce useful informa-tion such as symmetry properties, cord lengths, etc.

This approach and its variants can be used for retrieval of primitive shapes.The disadvantages of this approach are that it is not stable to mesh connec-

tivity changes, it requires a pose estimation and has a spherical representation(with a sur-representation around the two poles).


Example queries are now presented for both the EGI Oct and EGI Sphere.

49

Query 1 2 3


Figure 5.13: EGI Oct (Princeton Dataset)

The EGI Oct algorithm retrieves a couple of similar objects, however it alsohas a few not so similar objects. This is a less reliable method.


Figure 5.14: EGI Oct (SCULPTEUR Dataset)

The EGI Oct performs much better on the SCULPTEUR Dataset with onlyone unexpected result, however it is still generally the right shape.

50

Query 1 2 3


Figure 5.15: EGI Sphere (Princeton Dataset)

The EGI Sphere algorithm performs slightly better, with only the microscopeobject being particularlu out of place due its shape.


Figure 5.16: EGI Sphere (SCULPTEUR Dataset)

The EGI Sphere method performs well on the SCULPTEUR Dataset willno unexpected results.

51

5.5 The 3D Hough Matching Module

Algorithm By: Tony TungName: HoughOct, HoughSphereID: 1092, 1090


This module allows retrieval of similar 3D objects based upon the 3D Houghdescriptor, which describes the triangle normal orientation.

This module may used for queries on complex objects. Meshes should haveunified normals.


Module Speed: FastModule Accuracy: High


Generate: This module has no generation parameters.Compare: This module also takes a string parameter specifying which dis-

tance metric to use (default is “euclidean”) and a real value asan optional argument to the distance metric.


The 3D Hough transform produces feature vectors which consist of 3D his-tograms of the parameters of the polar representation of the 3D mesh trianglesconsidered as planar surfaces. These parameters are weighted with the area andthe orientation of the associated triangles.

This approach requires a PCA pre-processing. It shows good results on theMPEG-7 database, and is stable to variations in the 3D mesh representation.

52


Query 1 2 3


Figure 5.17: Hough Oct (Princeton Dataset)

The Hough Oct algorithm (Figure 5.17) has a lot of quite dissimilar objects inthis example.


Figure 5.18: Hough Oct (SCULPTEUR Dataset)

The Hough Oct algorithm (Figure 5.18) performs much better on the SCULP-TEUR dataset with all objects retrieved being similarly shaped.

53

Query 1 2 3


Figure 5.19: Hough Sphere (Princeton Dataset)

The Hough Sphere (Figure 5.19) algorithm has a lot of quite dissimilar ob-jects in this example.


Figure 5.20: Hough Sphere (SCULPTEUR Dataset)

The Hough sphere (Figure 5.20) performs slightly better with the SCULP-TEUR dataset, however it still has several unexpected results.

54

Chapter 6

Distance Metrics andNormalisation

6.1 Minkowski Norms

This Minkowski norms are some of the most popular distance metrics used. Thisis because they fast to compute and manage to produce quite reasonable resultsgiven their simplicity. The L1 and L2 norms are typically used. These are oftenknown as the city block and Euclidean distance respectively.

The main problem with the norms is that they treat each component sepa-rately. This assumes there is no correspondence between histogram bins.

D (X, Y ) = N

√∑|x − y|N

6.2 Histogram Intersection

Arguably one of the simplest distance metrics, the histogram intersection isoften used due to its speed rather than quality of results.

D (X, Y ) =∑

x −∑

min (x, y)

6.3 Bhattacharyya Distance

The Bhattacharyya distance is used to compare the similarity between two his-tograms. It is however quite slow due to the square root operations.

D (X, Y ) =∑

x −∑√

x√

y

6.4 Kullback-Leibler Divergence

The Kullback-Leibler divergence is a measure often used in information theory.Two forms are defined as symmetric version (as are all the other distance metrics

55

described here) and a non-symmetric version.Symmetric:D (X, Y ) =

∑(x − y) ln x

yNon-symmetric:D (X, Y ) = x ∗ log

(xy

)6.5 Chi-Squared

The χ2 test is the formal statistical method for assessing the dissimilarity be-tween two probability distributions. The distance metrics uses the χ2 divergencevalue as the full test is not required to get a dissimilarity score.

D (X, Y ) =∑ (x−y)2

x+y

6.6 Quadratic Distance

This is similar to the Euclidean distance, except it also makes use of a similar-ity matrix which aims to take into account correspondence between histogrambins. How this matrix is defined is typically dependant on the application. InSCULPTEUR the similarity matrix is built using the distance between bins.

D (X, Y ) =∑

i

∑j (xi − yi) · Aij · (xj − yj)

where Aij = exp−σ·(i−j) and i, j are the indexes to the histogram bins.

6.7 Normalisation

Normalisation is provided through a wrapper module. The module stores ahistogram of dissimilarity scores within a specified range (default [0.0:1.0]). Bydefault there are 1000 histogram bins. The module normalises a given scorebased upon its histogram of scores and returns a new score between 0.0 and 100.0where 100.0 is a perfect match. This corresponds roughly to the probability ofnot getting a better match than the query. Normalisation data is stored in ablob in the same method as feature vectors. The normalisation factor is updatedby adding new scores to the histogram.

To add a score to the histogram, we first take the log of the score. We add1 to the score first to avoid problems with taking the log of a number less than1.0. We then divide the score by the maximum score specified and multipliedby the number of histogram bins. This gives an index into the normalisationhistogram. If we are updating the histogram, then we increment the bin value.If we are normalising the score, we need to normalise the histogram so that thelargest bin has a value of 100.0. The normalised score is then 100.0 minus thevalue of the corresponding bin.

A good distribution of scores is required. So in the ideal case, a compari-son of each object against every other object should be used. Adding further

56

comparisons e.g. from query objects is less useful as it may start to skew thenormalisation away from the reference dataset.

57

Bibliography

[1] MySQL AB. Mysql. http://www.mysql.com [Accessed 2005-05-06].

[2] Matthew Addis, Mike Boniface, Simon Goodall, Paul Grimwood, SangheeKim, Paul Lewis, Kirk Martinez, and Alison Steveson. SCULPTEUR:Towards a New Paradigm for Multimedia Museum Information Handling.In International Semantic Web Conference (ISWC 2003), pages 582–596,Florida, USA, October 2003.

[3] Matthew J. Addis, M. J. Boniface, Simon Goodall, Paul Grimwood,Sanghee Kim, Paul Lewis, Kirk Martinex, and Alison Stevenson. Inte-grated image content and metadata search and retrieval across multipledatabases. In E. M. Bakker, T. S. Huang, and M. S. Lew, editors, Sec-ond International Conference on Image and Video Retreival 2003, pages91–100, Urbana-Champaign, Illinois, USA, 2003.

[4] S. Chan, K. Martinez, P. Lewis, C. Lahanier, and J. Stevenson. Handlingsub-image queries in content-based retrieval of high resolution art images.Proceedings of International Cultural Heritage Informatics Meeting, 2:157–163, September 2001.

[5] Web 3D Consortium. The virtual reality modeling language.online - http://www.web3d.org/x3d/specifications/vrml/ISO IEC 14772-All/index.html. Access 01/09/2004.

[6] John Cupitt, Kirk Martinez, and Joe Padfield. Vips.http://www.vips.ecs.soton.ac.uk/ [Accessed 2005-05-06].

[7] M. F. A. Fauzi and P. H. Lewis. Texture based image retrieval using mul-tiscale sub-image matching. In Proceedings of SPIE Conference on Imageand Video Communications and Processing, pages 407–416, 2003.

[8] Simon Goodall, Paul Lewis, and Kirk Martinez. 3-D shape descriptors anddistance metrics for content-based artefact retrieval. In R. W. Lienhart andN. Babaguchi, editors, Proceedings of Storage and Retrieval Methods andApplications for Multimedia 2005, pages 87–97, San Jose, California, USA,January 2005.

58

[9] Simon Goodall, Paul. H. Lewis, Kirk Matrinez, Patrick A. S. Sinclair, Fab-rizo Giorgini, M. J. Addis, M. J. Boniface, Christian Lahanier, and JamesStevenson. SCULPTEUR: Multimedia Retrieval for Museums. In Imageand Video Retrieval: Third International Conference, CIVR 2004), pages638–646, Dublin, Ireland, July 2004.

[10] Independant JPEG Group. Jpeg library. http://www.ijg.org [Accessed2005-05-06].

[11] Satoshi Konno. Cyber x3d. http://www.cybergarage.org/vrml/cx3d/cx3dcc/index.html[Accessed 2005-05-06].

[12] Paul H. Lewis, Kirk Martinez, Fazly Salleh Abas, Mohammad Faizal Ah-mad Fauzi, Stephen C. Y. Chan, Matthew J. Addis, Mike J. Boniface,Paul Grimwood, Alison Stevenson, Christian Lahanier, and James Steven-son. An Integrated Content and Metadata Based Retrieval System for Art.IEEE Transactions on Image Processing, 13(3):302–313, March 2003.

[13] G. Pass, R. Zabih, and J. Miller. Comparing images using color coherencevectors. In Proceedings of ACM Multimedia, pages 65–73, 1996.

[14] The GNU Project. Autoconf. http://www.gnu.org/software/autoconf [Ac-cessed 2005-05-06].

[15] The GNU Project. Automake. http://www.gnu.org/software/automake/[Accessed 2005-05-06].

[16] GCC steering committee. Gnu compiler collection. http://gcc.gnu.org [Ac-cessed 2005-05-6].

[17] Michael J. Swain and Dana H. Ballard. Color Indexing. InternationalJournal of Computer Vision, 7(1):11–32, 1991.

[18] Frank Warmerdam, Andrey Kiselev, and Mike Welles. Tiff library.http://libtiff.org [Accessed 2005-05-06].

59

Documents

FVS: A Content-Based Retrieval Librarylibfvs.sourceforge.net/docs/fvs-manual-1.0.2.pdf · FVS 3.1 Introduction FVS is the name of the library providing the 2-D image and 3-D object