Invariant Shape Features and Relevance Feedback for Weld Defect Image Retrieval

Nafaa NACEREDDINE 1, Djemel ZIOU 2

1 Centre de Recherche en Soudage et Contrôle, DTSI; Algiers, Algeria Phone/Fax: +213 21 361850; E-mail: nafaa.nacereddine@enp.edu.dz

2 DMI, Sherbrooke University; Quebec, Canada; E-mail: djemel.ziou@usherbrooke.ca

Abstract Relevance feedback mechanism is used in Content-based Image Retrieval (CBIR) to attempt to minimize the amount of interaction between the user and the system required to improve the retrieval system performance. In this work, such system is proposed for weld radiograms in radiographic testing, with the aim of searching from the overall image database, interactively with the radiograph expert, discontinuities similar to some common weld defect types such as, crack, lack of penetration, porosity and solid inclusion. Similarity measures use feature vectors based on shape descriptors invariant to usual geometric transformations. Experiments over the tested database demonstrate that the CBIR gives good results and is practical and promising for the future of welded joint radiographic examination. Keywords: Weld defect, radiographic testing, shape descriptor, CBIR, relevance feedback 1. Introduction Image analysis and pattern recognition techniques in non-destructive radiographic inspection are being increasingly used to increase the objectivity, consistency and efficiency of weld examination. Developing computer-aided radiogram interpretation system comprises three main steps: (1) use of adequate segmentation technique to detect the discontinuity, (2) selection of discriminating features to characterize it and (3) construction of accurate classifier or retrieval system for its identification. The purpose of the first part of this paper is then to develop effective shape descriptors permitting to characterize the weld defects in the radiographic image so that they are recognized by the image retrieval system as elements of defect classes easily identifiable. To this end, two shape descriptors are proposed. The first one called shape geometric descriptor (SGD) contains shape geometric measures (compactness, elongation, symmetry, etc.) derived from some object geometric parameters (area, inertia axes, equivalent ellipse, etc.). The second one is the well known generic Fourier descriptor (GFD) [1]. In radiographic testing, the possibly found weld defect may have different resolutions and orientations because it can be viewed from different angles according to the orientation and the distance of the irradiated welded component w.r.t.1 the radiation source site. This is why, the chosen descriptors, to be effective, must at first be invariant w.r.t. usual geometric transformations of translation, rotation and scaling. Many works were devoted to defect classification in weld radiographic inspection using various approaches [2−8]: artificial neural network (ANN), support vector machine (SVM), fuzzy logic, k-nearest neighbor (k-NN) , etc. However, all the classifiers have as objective to divide the overall image database into weld defect classes, requiring sometimes considerable execution time, especially with the presence of large image database. When inspecting welds for some specific industrial realizations, we are sometimes interested by the occurrence of a particular type of weld defects, for many reasons such as the relationship between the appearance frequency of such defect and the used welding process or material. For this reason, using a content-based image retrieval (CBIR) technique in this area where the subject 1 w.r.t. means with respect to

6th NDT in Progress 2011 International Workshop of NDT Experts, Prague, 10-12 Oct 2011

is rarely addressed [9], a particular defect can be systematically retrieved from the weld radiographic image collection and the required precautions or measures can be taken concerning the implicated welded realization. In this work, the investigated image database consists of radiographic images presenting indications of crack, lack of penetration, porosity and solid inclusion. Content-based Image Retrieval (CBIR) which is an actively researched area in computer vision is a process to find images similar in visual content to a given query from an image database. The CBIR system performance can be improved by using relevance feedback (RF) mechanism which supports human-computer interaction and where the user’s knowledge and feedback are used to refine the retrieval results. 2. Feature extraction 2.1 Shape geometric descriptor (SGD) Given an object image I = f(x, y); 0 ≤ x ≤ M, 0 ≤ y ≤ N and RI the image region occupied by the object, then f(x,y) = 1 if (x, y) ∈ RI and f(x, y) = 0 elsewhere. In our application, this object represents the weld defect, on which some geometric parameters can be measured: Area (A), perimeter (P), centre of mass )y,xG( , orientation angle (α), principal inertia axes, width (W) and length (L) of the minimal bounding box, partial surfaces of symmetry (S1),(S2),(S3) and (S4), major and minor axes (a,b) of the equivalent ellipse (see Fig. 1).

Figure1. Illustration of some shape geometric parameters From the shape geometric parameters and the polar signature, we define a set of shape geometric descriptors (SGD) which are invariant to common geometric transformations of translation, rotation and scaling.

• Compactness : 24 PAC π= . It represents the ratio of the shape area to the area of a circle (the most compact shape) having the same perimeter

• Elongation : WLE = . It is defined by the ratio of the length to the width of the minimal rectangle surrounding the object called also the minimal bounding box

• Rectangularity : WLAR ×= . The rectangularity degree R is equal to the ratio of the shape area to the area of its minimal bounding box:

• Eccentricity : ab−=1ε . The eccentricity degree ε is obtained from the ratio of the minor axis to the major axis in the object equivalent ellipse.

• Symmetry : VH SymSymSym ×= where SymH and SymV are respectively, the

horizontal and vertical symmetries, expressed by:

( ) ( )( ) ( )

++++=++++=

41324132

21432143

,sup/,inf

,sup/,inf

SSSSSSSSSym

SSSSSSSSSym

H

V

• Cumulative signature : ( )( )( ) ( )∫=π

θθθθπ

2

0max21 drrSig .

The shape signature is a 1-D functional representation of the boundary and can consist simply in the distance from the center of mass to the object boundary in function of the angle as illustrated in Fig. 1. The polar signature is invariant to translation but depends of rotation and scaling. We propose to integrate the normalized signature (w.r.t. 2π× its maximum) on 2π, in order to make it also invariant to rotation and scaling

We give in Table 1. relationships between SGD and defect types. See [2] for more details.

Table 1. Relationships between weld defect types and SGD

Attribute Values CR LP PO SI

C [0 1] → 0 < 0.5 → 1 → 0.5

E ≥ 1 >> 1 >> 1 → 1 > 1

R [0 1] < 0.5 → 1 > 0.5 → 0.5

ε [0 1] → 1 → 1 → 0 → 0.5

Sym [0 1] < 0.5 > 0.5 > 0.5 < 0.5

Sig [0 1] < 0.5 < 0.5 → 1 > 0.5

2.2 Generic Fourier descriptor (GFD) Fourier transform has been widely used for image processing and analysis. The advantage of analyzing image in spectral domain over analyzing shape in spatial domain is that it is easy to overcome the noise problem which is common to digital images. At first a polar transformation of an input image f(x,y) is done, obtaining a polar image f(r,θ) by

( ) ( ) 22 yyxxr −+−= ;xx

yy

−−= arctanθ (1)

where ( )yx, is defined in §2.1. Then a transformation of polar raster sampled image in Cartesian space is created by circular sampling of an object or an area in an image. The rotation of an image in Cartesian space results in a circular shift in polar space. According to the translation properties of the Fourier transform, a shift in the spatial domain results in a phase change in the frequency domain. By only retaining the magnitudes of the Fourier coefficients, then rotation invariance can be achieved, as sown in Fig. 2. Translation invariance is achieved by using the center of mass as the origin in polar space. The discrete Fourier transform (DFT) of the obtained polar-raster sampled image is then computed as

Figure 2. (a) & (d) Object and π/2-rotated object, resp.; (b) & (e) polar raster sampled image of (a) & (d), resp. ; (c) & (f) Fourier transform spectrum of (b) & (e), resp.

( ) ( )

+−= ∑∑ φρπθφρT

i

R

rjrf

r ii 2exp ,,PF (2)

where 0≤ ρ<R and θi = i(2π/T ); (0≤ i<T) ; 0≤ ρ<R , 0≤ φ <T. R and T are the resolutions of radial and angular frequencies, respectively. The generic Fourier descriptor [1] is computed from the normalized Fourier coefficients and are given by

×=

)0,0(

),(,,

)0,0(

)0,(,,

)0,0(

),0(,,

)0,0(

)1,0(,

)0,0(GFD 2

max PF

nmPF

PF

mPF

PF

nPF

PF

PF

r

PFLLL

π (3)

where m and n are, respectively, the maximum numbers of the radial and angular frequencies selected and rmax is the maximum radius of the object. The first element of the vector is normalized by the area of the circle in which the polar image resides and the others by the value |PF(0, 0)|. Then GFD becomes, in addition, invariant to scaling factor. 2. CBIR and Relevance feedback The CBIR process consists in finding images similar in visual content to a given query from an image database. It is usually performed based on a comparison of low level features, such as color, texture or shape features, extracted from the images themselves [10]. Nevertheless, there is an important mismatch between the capabilities of CBIR techniques and the needs of the users. This problem is called semantic gap which has been extensively used in the CBIR research community to express the discrepancy between the low-level features that can be readily extracted from the images and the descriptions that are meaningful for the users. Iterative refinement of the retrieval process by relevance feedback (RF) is one of mechanisms for reducing the semantic gap in CBIR technology by including the user in the retrieval loop [11]. Generally speaking, previous relevance feedback methods can be classified into two approaches: the “feature reweighting” approach and the “statistical” approach (e.g., [12]). The former, which is subject of our interest, is based on the mathematical criterion optimization (e.g., [13-15]). These approach techniques are one of the most popular formulations for relevance feedback in which, each feature component (or feature vector) is associated with a

weight. Once the weights are determined by the learning methods, these weights are then employed to measure the image distance using a weighted scheme. MindReader [13] integrated the two independent processes (query refinement and feature reweighting) into a unified framework, based on the minimization of total distances of positive example (PE) images from the new query, in order to derive the ideal query and feature weights theoretically. Furthermore, Rui et al. [14] extend MindReader’s work using a hierarchical distance model and define the image distance as a combination of multiple feature distances to better deal with singularity issue due to the small number of training samples. In [15], irrelevant feedbacks are also included to refine the learning process. MindReader and Rui model are used in this paper for the purpose of the comparison between the shape descriptors mentioned above and their combined version, as it will be detailed later. Here, we give the theoretical framework of these feature reweighting methods. For this purpose, we review the formulation proposed by [14] and explain the difference between the formulations in [13] and [14]. Let each image in the database be decomposed into a set of I features, each of which is a vector of real. Let nix

rbe the i th feature vector of the nth training or

PE image gained from user feedback and the N element vector [ ]Nπππ ,,1 Lr = represents the

degree of relevance (to the query iqr

) of each of the N positive feedback images, which can be

determined by the user at each feedback iteration. Each individual feature’s similarity is modeled as a generalized Euclidean distance because of its powerfulness and the overall similarity is modeled as linear combinations of each individual feature’s similarity because of its simplicity. We define the global dispersion of the query images by (4), where T denotes matrix transposition and for each feature i, Wi is an ellipsoidal distance matrix and ui a scalar weight. By minimizing the dispersion J of the training example images, we try to enhance the concentrated features, i.e., those features for which the PE images are close to each other.

( ) ( )ini

N

ni

Tinin

I

ii qxWqxuJ

rrrr −−= ∑∑== 11

π (4)

Furthermore, to avoid numerical stability problem, we introduce the following constraints:

∑=

=I

iiu

1

11 and det(Wi) = 1; i = 1,…, I

The optimality of the objective function J will be achieved by minimizing the distances between the ideal query and the entire positive fed back examples. The degree of relevance πn of each example is given by the user according to his or her judgment. The objective function J is linear in u

r and Wi and quadratic in iq

r. Again, we use Lagrange multipliers to solve a

constrained optimization problem, by computing iqr

, Wi and ui which minimize the quantity L

given by:

( )( )∑∑==

−−

−−=

I

i

ii

I

i i

Wu

JL11

1det11 λλ (5)

By setting iqL

r∂∂ / to zero, we can obtain the optimal solution for iqr

as

∑ ==

N

nni

TTi Xq

1

* ππrr (6)

where Xi is the N × Ki (Ki is the length of feature i) training sample matrix for feature i

obtained by stacking the N positive feedback training vectors Tnixr

into a matrix. The above

equation corresponds to the intuitive formulation of a query vector, which can computed for each feature i, as the weighted average of all training or PE images. To obtain the optimal solution of Wi, we consider the partial derivative of L w.r.t

rsiw , where

][rsii wW = and r, s = 1, … , Ki ; ( )( ) ( ) ( ) 0det1

1

=−−−−=∂∂ +

=∑ rsssrr

rs

isr

i

N

n

iniininii

Wqxqxuw

L λπ where

( ) ( )( ) ( ) sri

N

niniininii ssrrrs

qxqxuW +

=−−−= ∑ 1det

1

λπ with ( )( ) ∑∑==

−−=N

n

n

N

n

iniinini ssrrrsqxqxC

11

ππ

After computing 1−iW the inverse matrix of Wi, and using the constraint det(Wi) = 1, the

optimal solution for Wi is given by

( )( ) 1/111* det −−− == i

K

iii CWCW iγ (7)

where ( ][rsii cC = ) is the weighted covariance matrix of Xi and

= ∑ =

N

nnii u

1πλγ

In a similar manner, the optimal solution for ui is computed as

∑=

=I

j i

ji f

fu

1

* (8)

where ( ) ( )∑ =−−= N

n iniiT

inini qxWqxf1

rrrrπ .

This solution can be interpreted as follows: high feature weight value corresponds to feature leading to low total distance, i.e. to feature with high discrimination ability. In MindReader model, each image is represented by one single feature vector consisting of all the adopted feature components, where the feature weight ui is omitted. That is why, this model can be referred as “flat model”, conversely to the Rui’s model which can be referred as ”hierarchical model” where the overall distance is defined as a weighted combination of individual feature distances. By using the Lagrange multipliers, analogously to Rui’s model,

the optimal solutions for MindReader can be derived: ∑ == N

n nTT Xq

1

* ππrrwhere X is the

N×K (K is the length of feature vector) training sample matrix obtained by stacking the N

positive feedback training vectors Tnxr

into a matrix; whereas, ( )( ) 1/111* det −−− == CWCWIγ

where C is the weighted covariance matrix of X. For both models, the computation of the weight matrix Wi (resp. W) requires the inversion of the matrix Ci (resp. C). It is clear that, if N < Ki (resp. N < K), then Ci (resp. C) is not invertible and Wi (resp. W) cannot be obtained. To overcome this problem, the authors in [14], when N < Ki, to replace Wi by a diagonal matrix whose elements are the inverse of the standard deviation, i.e.,

rsiw = 1/ri

σ if r = s and rsiw = 0, elsewhere, and this, to ensure reliable

estimation. In this paper, this weight matrix calculation scheme is adopted, but by replacing, in the case of N < Ki, the standard deviation

riσ by the diagonal element

rric of the covariance

matrix (Ci), as proposed by Kherfi et al. [15].

3. Experimental results More than three hundred weld defect regions are extracted from weld radiographic films provided by the International Institute of Welding (IIW) which represent four weld defect classes: crack (CR), lack of penetration (LP), porosity (PO) and solid inclusion (SI). After the segmentation step is achieved where the weld defect region is represented by its binary version [16], the database is indexed by the proposed descriptors, namely, SGD and GFD. Some of these defects are illustrated in Fig. 3.

Figure 3. A sample of weld defect regions and their binarized versions CBIR system with RF is carried out on the weld defect image database mentioned above as shown in Fig. 4. Our objective by applying the proposed framework is to respond to the following preoccupation: given a defect or a set of defects belonging to certain weld defect class, we should retrieve all defects belonging to this class from the entire database. To validate MindReader and Rui’s relevance feedback models in the context of our application, a query chosen from the database, representing the searched weld defect image, is formulated to the CBIR system. A set of images is then returned to user, ranked according to their similarity to the query. These preliminary results do not utilize the relevance feedback. This step is noted by Rf0. Some relevant images are chosen by the user from the returned images and fed back to the system as training examples after assigning the corresponding relevance degrees. The possible relevance degrees are: very similar: πn = 2; similar: πn = 1; does not matter or different: the image will not participate in the query. Once the query formulation is completed, the user can launch the retrieval process. The system computes the optimal parameters for Minreader and Rui’s model and performs a distance-based comparison and the database images are ranked according to their closeness to the query. This distance is given, for MindReader, by

( ) ( )*** qxWqxD m

T

mmMR rrrr −−= (9)

whereas, it is formulated for Rui’s model by

( ) ( )*

1

***imi

I

ii

T

imiimRui qxWqxuD

rrrr −−= ∑=

(10)

where, m = 1 ,…, M with M is the number of images in the test database. Thus, the first relevance feedback sequence Rf1 is completed. The top-ranked images are returned to the user by the system. However, he/she can ask to see more images in order to feed back the new query. If the user is not satisfied with the retrieved images, a second relevance feedback sequence Rf2 will be performed, in the same manner as Rf1, and so on.

Figure 4. Synoptic diagram of the proposed weld defect image retrieval with relevance feedback

To perform any evaluation in the context of image retrieval, two major issues must be addressed: the acquisition of ground truth and the definition of performance criteria. To construct the ground truth, the overall weld defect image database is divided a priori, by a radiograph expert, in four image sets representing the weld defect classes mentioned in §3. Concerning performance criteria, those most often used are precision (Pr) and recall (Re) [17]. In their simplest definition, precision is the proportion of retrieved images that are relevant, i.e., the number of retrieved images that are relevant over the total number of retrieved images; and recall is the proportion of relevant images that are retrieved, i.e., the number of relevant images that are retrieved over the total number of relevant images in the database. In this paper, we draw up the precision-recall curve Pr = f(Re). The retrieval results averaged on the four defect classes for SGD, GFD, concatenation of SGD and GFD (SGD+GFD), and Rui (SGD,GFD) without and with RF are given in Fig. 5. We can notice that firstly, the Rui model which combines SGD and GFD in a hierarchical way outperforms MindReader model with SGD and GFD used separately or simply concatenated. Secondly, the retrieval precision is improved considerably in the feedback case than that in the non-feedback case. The precision increases with successive iterations of RF but the largest

improvement is achieved with the first iteration. This is property is very interesting, since it permits an achievement of an acceptable retrieval results within a limited amount of RF cycles. An example is illustrated in Fig. 6 where it is shown the insufficiency of GFD to discriminate between a crack and a lack of penetration for Rf0 step. Nevertheless, one relevance feedback sequence is sufficient to improve the retrieval results.

Figure 5. Retrieval results averaged on all defect classes without RF and with 1st and 2nd RF iterations

Figure 6. Retrieval results for GFD without RF (Rf0) and after 1st sequence of RF (Rf1) 3. Conclusion To characterize defects issued from weld radiographic examination, in terms of their shapes, we have proposed in this paper, two descriptors namely SGD and GFD. Whereas, the components of SGD are selected among many others on the basis of their shape discrimination capacities and their direct relationships to the studied weld defects types, GFD is chosen not for our image characteristics, but for its well known powerfulness for the filled shape description. In weld radiographic inspection, we are sometimes interested by the occurrence of a certain defect type without being concerned with the other types. Content-based image retrieval can answer this requirement. In such a system, it is not necessary to divide the overall database images into classes or categories as it is the case in classification, because only one subset of this database is searched. A particular defect type can be tracked for various reasons such as: particular evaluation of welding process, degree of harmfulness for the industrial component in question, etc. Moreover, CBIR with relevance feedback can give more realistic approach for weld defect identification than the one given by classification because the intervention of the radiography expert is very significant in interpretation; the task, which can be accomplished through relevance feedback.

0 10 20 30 40 50 60 70 80 90 10070

75

80

85

90

95

100

Recall %

Pre

cisi

on %

1st sequence of RF (Rf1)

SGD

GFDSGD+GFD

Rui(SGD,GFD)

0 10 20 30 40 50 60 70 80 90 10070

75

80

85

90

95

100

Recall %

Pre

cisi

on %

Without RF (Rf0)

SGD

GFDSGD+GFD

Rui(SGD,GFD)

0 10 20 30 40 50 60 70 80 90 10070

75

80

85

90

95

100

Recall %

Pre

cisi

on %

2nd sequence of RF (Rf2)

SGD

GFDSGD+GFD

Rui(SGD,GFD)

References 1. D Zhang and G Lu, ‘Shape-based image retrieval using generic Fourier descriptor’,

Signal Processing, Vol 17, No 10, pp 825-848, 2002. 2. N Nacereddine and M Tridi, ‘Computer-aided shape analysis and classification of weld

defects in industrial radiography based invariant attributes and neural networks’, 4th Int. Symposium on Image and Signal Processing and Analysis, Zagreb, pp 88-93, 2005.

3. K Aoki and Y Suga, ‘Application of artificial neural network to discrimination of defect type automatic radiographic testing of welds’, ISI International Vol 39, No 10, pp 1081-1087, 1999.

4. S V Baraia and P Agrawal, ‘Parallel Neuro Classifier for Weld Defect Classification’, Advances in Soft Computing Vol 34, pp 37-54, 2006.

5. T Y Lim, M M Ratnam and M Khalid, ‘Automatic classification of weld defects using simulated data and an MLP neural network’, Insight, Vol 49, No 3, pp 154-159, 2007.

6. I Valavanis and D Kosmopoulos, ‘Multiclass defect detection and classification in weld radiographic images using geometric and texture’, Expert Systems with Applications, Vol 37, No 12, pp 7606-7614, 2010.

7. X G Zhang, J J Xu and G Y Ge, ‘Defects recognition on X-ray images for weld inspection using SVM’, Int. Conf. on Machine Learning and Cybernetics, Vol 6, China, pp 3721-3725, 2004.

8. T W Liao, ‘Classification of welding flaw types with fuzzy expert systems’, Expert Systems with Applications, Vol 25, No 1, pp 101-111, 2003.

9. N Nacereddine, L Hamami, D Ziou, A B Goumeidane and F Mekhalfa, ‘Content-Based weld defect image retrieval in radiographic testing’, 6th Conf. on NDE in Relation to Structural Integrity for Nuclear and Pressurized Components, Budapest, 2007.

10. Z Su, H Zhang, S Li and S Ma, ‘Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces, and Progressive Learning’, IEEE Trans. on Image Processing, Vol 9, No 8, pp 924-937, 2003.

11. R Missaoui, M Sarifuddin and J Vaillancourt, ‘Similarity measures for efficient content-based image retrieval’, IEE Proc. - Vision, Image and Signal Processing, Vol 152, No 6, pp 875-887, 2005.

12. I J Cox, M L Miller, T P Minka, T V Papathomas and P N Yianilos, ‘The bayesian image retrieval system, PicHunter: Theory, implementation and psychophysical experiments’, IEEE Trans. on Image Processing, Vol 9, No 1, pp 20-37, 2000.

13. Y Ishikawa, R Subramanya and C Faloutsos, ‘Mindreader: Query databases through multiple examples’, In Proc. 24th Intern. Conference on Very Large Databases, NY, USA, pp 218-227, 1998.

14. Y Rui and T S Huang, ‘Optimizing learning in image retrieval’, in IEEE Conf. on Computer Vision and Pattern Recognition, Vol 1, SC, USA, pp 236-243, 2000.

15. M L Kherfi, D Ziou and A Bernardi, ‘Combining Positive and Negative Examples in Relevance Feedback for Content-Based Image Retrieval’, Journal of Visual Communication and Image Represention, Vol 14, No 4, pp 428-457, 2003.

16. N Nacereddine, L Hamami, D Ziou, ‘Thresholding techniques and their performance evaluation for weld defect detection in radiographic testing’, Machine Graphics and Vision, Vol 15, No 3-4, pp. 557-566, 2006.

17. J R Smith, ‘Image retrieval evaluation’, Proc. of the IEEE Workshop on Content-based Access of Image and Video Libraries, 1998.