NAFIPS 2005 - 2005 Annual Meeting of the North American Fuzzy Information Processing Society

DESIGN OF A VISION SENSOR USING FUZZY ASSOCIATIVE DATABASE

Xiang Chent*, Shahed Shahirt, and Majid Ahmadi

ABSTRACTIn this paper a design method is proposed for apotential new camera-based intelligent vision sen-sor. This sensor can be used for fast multiple pla-nar object recognition. The mechanism behindthe design is a Fuzzy Associative Database (FAD)which consists of a Fuzzy Database (FD) and aFuzzy Search Engine (FSE). The FSE uses tableone to conduct search over table two , both inFD, through a Bank of Fuzzy Associative Mem-ory MIatrix(BFAMM). In fact, the FSE establishesa correspondence between an object and one ofthe trained classes in table two. Therefore, theFAD could actually 'remember' the trained ob-_jects and the FSE could 'recognize' the incomingobject by comparing it with trained informationin the database. The experimental results showthat this approach is robust and fast.

1. INTRODUCTION

Smart vision sensor design attracts a lot of aca-demic and industrial interests as many existingproduction problems require the inclusion of intel-ligent sensor solutions into the mianufacturing pro-cesses. The current research and design interestsconcentrate on hardware implementation and vi-sion processing algorithms (see [1, 4, 7, 11, 12, 22,18]). The optical hardware currently used in non-contact smart vision sensors include laser scan-ners, photo-diodes and cameras. Among them,

THIS WORK WAS SUPPORTED IN PART BY MA-TERIAL AND MANUFACTURING ONTARIO.

CORRESPONDING AUTHOR, E-MAIL:XCHEN©UWINDSOR.CAt: DEPARTMENT OF ELECTRICAL AND COMPUTERENGINEERING, UNIVERSITY OF WINDSOR, WIND-SOR, ONTARIO, CANADA N9B 3P4f: DEPARTMENT OF ELECTRICAL AND COMPUTERENGINEERING, UNIVERSITY OF WATERLOO, WA-TERLOO, ONTARIO, CANADA

the camera system captures the image informa-tion of objects through a non-scanning approachand, hence, potentially provides an effective infor-mation source for conducting real-time softwarebased position measurement. On the other hand,for post-sensing processing, in order to address theissues such as versatility and real-time applicationof the camera based vision sensor, robust and ef-ficient algorithms, in terms of object recognitionand computing load, must be investigated and de-veloped, originating from those for planar objectrecognitions.

Many algorithms for the purpose of planar ob-ject recognition have been developed so far. Amongthem, the template matching method is presentedin [?]; the correlation method has been appliedto obtaining a color motion stereo [16], automaticdefect classification [3], chemical structure match-ing [2], and template matching [17]; and the Di-rectional Flow Change(DFC) method is presentedin [23]. Vision sensor design based on correlationand DFC methods are also reported in [5, 6, 21].These methods are developed for recognizing a sin-gle object. Besides, it is relatively difficult to applythese algorithms to recognizing the rotation of pla-nar objects. A model-based recognition method isproposed in [20] to recognize planar objects un-der rotation but it is good only for regular shapessuch as square or rectangular shapes. The fuzzyinvariant indexing method developed in[8] is an-other way to handle object recognition.

The significance of the design method presentedin this paper is also echoed by a real engineer-ing problem existing in the foam barrier assem-bly line for automobile door handle escutcheons.The story is that the assembly process requires arobotic arm to locate the sheet foam barrier on aconveyor. However, one practical difficulty is thatthe position of foam barriers may deviate fromeach other due to possible stretching behavior of

0-7803-9187-X/05/$20.00 ©2005 IEEE. 286

foam material and that the pre-configured roboticarm cannot handle this position uncertainty be-cause there is no sensor involved in this process.On the other hand, this problem cannot be solvedby simply adding a traditional position sensor tothe robotic arm system because 1. finding the cor-rect position of foam barriers involves a searchingprocess which cannot be performed by the exist-ing sensors; 2. it is difficult to identify special fea-tures on the foam material as the position indica-tion. Clearly, new effective and economic solutionsneed to be figured out to accommodate such kindof engineering problems. One way is to develop anew smart vision sensor that possesses the capa-bility of measuring the position through recogniz-ing the patterns of objects and of 'remembering'the desired pattern of objects. Besides, the sensorshould be in modular form in the sense that it willbe easily re-configured to adapt for the changingproducts on the assembly line.

The proposed design method in this paper, asan initial design, tries to develop a sensor mech-anism that could be eventually evolved into sucha kind of smart vision sensor. The primary goalis to introduce a method to conduct planar objectrecognition in real world using the fuzzy databaseand searching techniques. This means that ourmethod allows the training and the rememberingof multiple objects, which is the key to the devel-opment of a robust and versatile vision sensor.

This paper is organized as following: in Sec-tion 2, FAD algorithm is summarized from [19];in Section 3, the idea of designing a smart visionsensor is stimulated by the position identificationof a sheet of foam barriers based on the FAD al-gorithm; the paper is concluded in Section 4 withfuture work stated.

2. FAD ALGORITHM

The design methodology for FAD is summarizedas follows:

1. transforming the object image to a binaryone,

2. determining the invariant values of the bi-nary image and fuzzify the values,

3. if the knowledge base for the target imagedoes not exist, the trainee will be added tothe knowledge base; while, if the knowledgebase does exist, the current object can be as-sociated to one of the trained object classesby applying the fuzzy search through thebase knowledge.

An FAD algorithm [19] is summarized for mul-tiple planar object recognition. With this algo-rithm, one first trains an FAD to remember tem-plates of multiple planar shapes and then the al-gorithm can be applied to automatic recognitionof multiple planar objects by associating the in-coming object to one of the trained classes. Thetraining of the FAD is based on invariant values ofan object template, while the recognition processis realized by the fuzzy search engine.

2.1. Invariant Values

Moment factor and compactness factor [7] are usedto characterize the invariant value of a binary im-age.

Definition 2.1 Let I, and I, be the second moment[15]of an image with respect to the X and Y axes givenby:

h w

= ES (Y -)2f (X, Y),x=l y=l

andh w

Iy E E(x _jS)2f(X, Y)x=1 y=l

where h and w are the height and the width ofthe image in pixels; x and y are the coordinates ofsingle pixel, and f(x, y) is the gray level value ofthe pixel at (x, y). The moment factor M of thisimage is defined as the Hamming norm [13] of thenormalized second moments:

M = Ix + Iy I

where Ix and Iy are the normalized second mo-ments given by

h w

E -(y_ 9)2f(x, y)

h w

E Ef(X,y)x=1 y-1

287

h w

E (x - t)2f(X, y)x=l y=l

h w

E Ef(X,y)x=1 y=l

2.3. Fuzzy Search Engine

A bank of fuzzy associative memory matrix (see[13]) is created to link the record of the first tableto the corresponding class in the second table ofthe fuzzy database in the following expression:

Definition 2.2 The compactness factor C of an im-age is defined as

h w 2

w EE fboundary (X,X)) 2C=I h w

E 5f(x,y)_=1 _1I

where fboundary(x, y) is the grey level of the pixelon the image boundary.

2.2. Fuzzy Database

Using the invariant values, a fuzzy knowledge data-base can be implemented consists of two tablesand holds the information about the known ob-jects while serving the memory function for recog-nizing different objects.

Definition 2.3 Given the moment factor Al andthe compactness factor C of an image, two fuzzysets F(z, a, k) induced by a = 34 and a = C sep-arately are called the fuzzified moment factor andthe fuzzified compactness factor of this image asdefined by:

F(z, a, k)= max (min( (1 k) (1 +k)a - z)Owhere z is a self-variable and k is a support factorcoefficient determined by the object thickness. Forexample, for the objects with the thickness lessthan one centimeter, k is 0.05.

If an object is under training in the fuzzy database,the fuzzified invariant values F(z, a, k) of the traineeare kept in the first table. Therefore, in general,the first table contains m records correspondingto m trained objects with 2 fields containing thefuzzified moment factor and the compactness fac-tor respectively for the corresponding trainee. Thesecond table is then indexed in the trained classesof objects, corresponding to the record in the firsttable.

BFAMM = fli f21f12 f22

f31 ... fml

f32 ... fm2 '

where fij(i = 1, 2,. . ., m), j = 1, 2 are the fuzzifiedinvariant values for the i-th trained class in table2.

Next a composition operation[10, 14] is definedas follows.

Definition 2.4 Let A be a 2 x 1 fuzzy vector and Bbe a 2 x m fuzzy matrix. Let aj, j = 1, 2 be thefuzzy entries of A and bij, i = 1,2, .. ., m; j =1, 2 be the fuzzy entries of B. Then the composi-tion operation returns a new 1 x m fuzzy vectorD = ATeB with the fuzzy entry di of D given by

2

di = I min(aj, bij), i = 1, 2, 3,... ., m.j=1

When an incoming object image is captured,a fuzzy vector(2 x 1) A can be obtained from thefuzzified moment factor M and the compactnessfactor C of this image. A fuzzy search through thefuzzy database can then be described by applyingthe composition operation E to A and BFAMMto result in a fuzzy vector D = ATEBFAMM, ofwhich the fuzzy entries di is given by

2

di = min(aj, fij), i = 1, 2, 3, .. ., rn.j=i

Note that the elements of the vector D show theextent of the similarity between the incoming ob-ject and the trained classes in the second table ofthe fuzzy database. The index i*, of the closesttrained class in the second table to the incomingobject is the one which maximizes di, that is

di.=maxdi, i=1.2...,n

Therefore the i*-th class in the second is the one

which determines the class of the incoming object.This fuzzy association process is shown in Figure1.

288

and

Iy =

Fuzzy Database

-

ITablel1Table 2

\tn a --|L----------

One

o _./~ ~ ~ ~ T

Fuzzy Search Engine

Fig. 1.

3. DESIGN OF A SMART VISION SENSOR

We shall be interested in designing a vision sensoras shown in the block diagram Figure 3.

Template

Image of Position

Fig. 2. Block Diagram of Smart Vision Sensor

In this configuration, a vision element (for ex-ample, a CCD camera)is needed to capture theimage of an object. A vision platform has beenbuilt to illustrate the sensor design idea (Figure??.

After preliminary processing of the image, abinary one is obtained. Then the FAD algorithmin Section 2 is kicked in to compare the imagewith the template stored in FAD which representsthe desired position of the object. The spatialshifts between the two images will be obtained andtranslated as the position deviations of the object.To explain the idea, we use an assembly line forsheet foam barriers on a conveyor as an exampleto show the algorithm design of the vision sensor.The image of a real sheet foam barrier on the con-veyor is shown in Figure 4.

Bv,dldL

a2

bl

Fig. 4. Image of Foam Barriers

The vision sensor is designed to identify thepattern of the real foam barrier on the conveyorby comparing the fuzzified invariant values of thereal image with the template image stored in thefuzzy database through the fuzzy searching andinferencing engine. To illustrate the design idea,the system is trained for four distinct classes: nor-mal, stretched, squeezed, and rotated foam barri-ers, respectively, shown in Figures 5 and 6. Then20 practical planar shapes of this foam barrier areapplied for test. All of them have been recognizedsuccessfully to be classified as normal, squeezed,stretched and rotated by the algorithm.

Fig. 3. Vision PlatformFig. 5.

289

Fig. 6.

4. CONCLUSION AND FUTURE WORK

It is illustrated that the FAD algorithm is a fea-sible approach for potential smart vision sensordesign to conduct multiple planar object recogni-tion especially when we are concerned about ro-tation. In general, as long as there exist distinctinvariant values for particular objects, the visionsensing recognition through the Fuzzy AssociativeDatabase can be achieved successfully. In future,research will be carried for the Learning Fuzzy As-sociation. This mechanism would be applied byadjusting the BFAMM based on the trainees tomake the vision sensor smarter.

5. REFERENCES

[1] 0. Appenzeller, P. Weckesser, R. Dillmann,"Active Parameter Control for the Low LevelVision System of A Mobile Robot", Proc.IROS, pp. 1256-1263, 1996.

[2] L. Austin, A. Turner, M. Turner, and K. Lees,"Chemical Structure Matching Using Corre-lation Matrix Memories", Proc. IEE Confer-ence on Artificial Neural Networks, pp. 619-624, 1999.

[3] J. Blais, V. Fischer, Y. Moalem, and M. Saun-ders, " Correlation of Digital Image Matricesto Production ADC Matching Performance",Proc. IEEE/SEMI Advanced SemiconductorManufacturing Conference, pp.86-92, 1998.

[4] Wai-Chi Fang, "A System-On-A-Chip DesignofA Low-Power Smart Vision System", Proc.1998 IEEE Workshop on Signal ProcessingSystems: Design and Implementation, pp. 63-72, 1998.

[5] Hongmei Gao and Xiang Chen, "TheoreticDesign of A Smart Vision Sensor", Proc. 2001IEEE Canadian Conference on Electrical andComputer Engineering, 2001, 1223-1228.

[6] Hongmei Gao, Xiang Chen and ZhangRen, "Algorithm Design for a Camera-BasedPosition Tracking Sensor Based on PatternRecognition", Proc. IEEE Industrial Elec-tronics Annual Conference 2002, pp. 2173-2178, 2002.

[7] Rafael C. Gonzalez,, "Digital Image Process-ing", Prentice-Hall, 2002.

[8] Thorsten Graf, Alois Knoll, and Andre Wol-fram, "Fuzzy Invariant Indexing: Ageneral In-dexing Scheme for Ocluded Object Recogni-tion", Proc. ICSP, pp. 908-911, 1998.

[9] P. W. Huang, S. K. Dai, and P.I. Lin, "Pla-nar Shape Recognition by Directional Flow-change Method", Pattern Recognition Let-ters, Vol. 20, No.2, pp.163-170,1999.

[10] Jyh-Sing Roger Jang, "Neuro-Fuzzy andSoft Computing: A Computational Ap-proach to Learning and Machine Intelli-gence", Prentice-Hall, 1997.

[11] G.K. Knopf, and S. Zhu, "Qualitative De-tection of Object Movement by Mobile Cam-era Systems", Proc. ISIAC Second Interna-tional Symposium on Intelligent Automationand Control, pp. 108.1-108.6, 1998.

[12] C. Koch, "Implementing Early Vision Algo-rithms in Analog Hardware-An Overview",SPIE, Vol.1473, pp.2-16, 1991.

[13] B. Kosko "Neural Network and Fuzzy Sys-tems", New Jersy:Prentice-Hall, 1992.

[14] Arun D. Kulkarni, "Computer Vision andFuzzy-Neural Systems", Prentice Hall, 2001.

[15] David J. McGill and Wilton W. King, "Engi-neering Mechanics Static", PWS PublishingCompany,1995

290

[16] M. Mozerov, V. Kober, and T.S. Choi, "ColorMotion Stereo Based on Adaptive Correla-tion Matching", SPIE, Vol.3808, pp. 693-701,1999.

[17] H. Penz, I. Bajla, K. Mayer, and W. Kratten-thaler, "High-Speed Template Matching withPoint Correlation in Image Pyramids", SPIE,Vol.3827, pp. 85-94, 1999.

[18] Tzung-Sz Shen, Jianbing Huang, and Chia-Hsiang Menq, "Multiple-Sensor Integrationfor Rapid and High-Precision CoordinateMetrology", Proc. the 1999 IEEE/ASME In-ternational Conference on Advanced Intelli-gent Mechatronics, pp. 908-915, 1999.

[19] Shahed Shahir, Xiang Chen, and Majid Ah-madi, "Fuzzy Associative Database for Multi-ple Planar Object Recognition", Proc. IEEEInternational Symposium on Circuits andSystems(ISCAS) , Vol. V, pp. 805-808, 2003.

[20] Humberto Sossa and Amparo Plomino,"M\odel-Based Recognition of Planar ObjectsUsing Geometric Invariants", IEEE, pp.603-606, 1996.

[21] Jun Yang and Xiang Chen, "A Real TimeAlgorithm for Planar Shape Recognition andPosition Deviation Analysis Based on Statis-tical and Directional Flow-Change Methods",Proc. IASTED International Conference onCircuits, Signals and Systems (CSS2003), pp.259-263, 2003.

[22] C. C. Yang, M. M. Marefat, and R. L.Kashyap, "Active Visual Inspection Based onCAD Models" , Proc. IEEE, pp.1120-1125,1994.

[23] P. W. Huang, S. K. Dai, and P.1. Lin, "Pla-nar Shape Recognition by Directional Flow-change Method", Pattern Recognition Let-ters, Vol. 20, No.2, pp.163-170,1999.

291