Color object identification by monochromatic binary correlation

Bahram Javidi, Chung-Jung Kuo, Ying Feng Chen, and Jacques E. Ludman

A real-time polychromatic image correlator that uses a magnetooptic (MO) spatial light modulator (SLM)device for pattern recognition based on both the color and shape of an input object is presented. Theproposed system utilizes a multichannel spectral matched spatial filter employed in a binary coherent opticalcorrelator. Input color images are transformed into binary color coded coherent images by a color grating.The color encoded images are read out by a charge coupled device interfaced with a MO SLM. The colorencoded binary images are then processed by a multichannel joint spectral matched spatial filter synthesizedby monochromatic light. Pattern recognition experiments for naturally illuminated real color objects arepresented.

1. Introduction

Optical pattern recognition systems using matchedfiltering techniquesl have been studied for many im-portant aplications. Their parallel operation at ex-tremely high speeds and with high resolution can be apowerful alternative to many digital or electronic com-puting techniques. Recent advances in the design ofspatial light modulators (SLMs) have led to the devel-opment of real-time optical pattern recognition sys-tems with potential applications in robotics, automaticinspection, identification, tracking, smart sensor, etc.

Existing SLMs such as the liquid crystal light valve2

(LCLV) and the liquid crystal televisions (LCTV) havebeen employed in various optical pattern recognitionarchitectures. Binary SLMs such as the magneto-optic (MO) device4 have been used in real-time opticalcorrelation by matched spatial filtering techniques.5

The majority of conventional coherent optical corre-lators perform signal detection based on shape aloneand they do not use the spectral content of the objectunder study. In real-time optical image processing,the color of objects can be as important as their shape.In some pattern recognition applications, it is possibleto improve the discrimination sensitivity of the systemby employing techniques that utilize both color andshape of the object under observation.6-9 For exam-

Jacques Ludman is with Rome Air Development Center, RADC/ESO, Hanscom Air Force Base, Massachusetts 01731; the otherauthors are with Michigan State University, Electrical EngineeringDepartment, East Lansing, Michigan 48824.

Received 16 October 1987.

ple, color encoding of objects in robotics and automaticinspection may facilitate the detection process andalleviate some of the problems associated with a robotsfalse detection.

In this paper we present a real-time polychromaticbinary pattern recognition system. Here, binary im-age correlation is performed based on the spectralcontent and shape of the input object. The systememploys a color encoding grating and a binary spatiallight modulator at the input plane to convert the inco-herently illuminated color object into spatially sam-pled binary coherent images. The spatially sampledbinary images are processed by a multispectral bandmatched spatial filter. The system introduced herehas two unique features that have not previously beenmet together: First, real-time image recognition isachieved based on both the spectral content and theshape of the object under study. Second, a CCD inter-faced binary spatial light modulator is used at theinput plane to peform a binary correlation of the inputcolor images. The binary image transducer used is aMO device. The sampled color objects are imagedonto a CCD array and the spatially encoded images arebinarized electronically using a predetermined thresh-old value. A MO device is used at the input plane toread out the thresholded color encoded images whichare then processed by a multispectral band matchedspatial filter.

There are real-time correlators that can performsignal detection based on the color and shape of theobject but they have different architectures and otherfeatures. The Yu et al. correlator, 9 for example, uses acolor LCTV as the input SLM and requires three lasersto illuminate the input SLM. The Ludman et al.correlator 8 uses a LCLV as the input SLM and requiresonly a single monochromatic laser. In this paper thecolor encoded input signals are binarized and the cor-

1 March 1988 / Vol. 27, No. 5 / APPLIED OPTICS 949

relator uses a binary SLM and requires only a singlemonochromatic laser. In this paper the color encodedinput signals are binarized and the correlator uses abinary SLM and requires only a single monochromaticlaser. It has been shown that binarizing the inputsignals and using the matched filter made from thebinarized input signal does not seriously degrade theperformance of the correlator and in some cases it canimprove the system performance.10 Thus, utilizingthis type of binarized correlation can result in im-proved color pattern recognition performance.

11. Analysis

The real-time polychromatic binary optical correla-tor is shown in Fig. 1. Plane P1 contains the colorobject illuminated by incoherent light, and plane P2contains the CCD and the color encoding grating. Thecolor encoded image detected by the CCD is binarizedelectronically by using a threshold network. Thethreshold value is determined by evaluating the histo-gram of the pixel values and picking the median. AMO device located at plane P 3 is used to read out thebinarized spatially encoded images by monochromaticlight. Plane P 4 is the Fourier plane with angular fre-quency coordinates (p,q) where the matched filter isinserted. Plane P5 is the output plane where the corre-lation signals are produced. Lens IL1 is an achromaticimaging lens, and lenses FTL 1 and FTL 2 are Fouriertransform lenses.

The color encoding grating consists of N subgratings(a red, a green, and a blue subgrating), and its functionis to encode every color of the input image with aspecific carrier frequency at a specific angular orienta-tion. Thus, the Fourier spectrum of the binarizedcolor encoded image is composed of N differently ori-ented diffracted spectra (Fig. 2) corresponding to eachspecific color (blue, green, and red). Polychromaticsignal detection can be achieved by using a multichan-nel color encoded matched spatial filter. This is doneby generating a matched filter for each color using thecorresponding diffracted spectra.

The color correlator described here is intended tooperate under a monochromatic readout light source.Therefore, each submatched filter corresponding tothe diffracted spectra is synthesized with a differentspatial carrier frequency as shown in Fig. 1. In thisfigure, b, g and 0

r are the modulating angles corre-sponding to blue, green, and red diffracted spectra.The submatched filter function for each color can bewritten as

Hn(pq;X0 ) = F(pq;x) exp[if (sino0 )p], n = 1,2,3, (1)

where On is the modulating angle corresponding to red,green, and blue spatial contents of the image, respec-tively, X is the wavelength of the monochromatic read-out light illuminating the MO device, and Fb(.) is theFourier spectrum of different binarized color encodedobjects. Since the matched filter of diffracted spectrais modulated at a different spatial carrier frequency,the output correlation consists of three different peaksat three different locations with the intensity of each

Color Grating CCD

LinearlyPolarizedCollimated

Light

MOD Polaizer

Reference Beams

IL: Imaging Lens

FTL: Pourier Tansform LeonMOD: Magneto-Optic Device

Fig. 1. Real-time polychromatic binary optical correlator.

Gren

Red Ble

Fig. 2. Diffracted Fourier spectra for blue, green, and red colors.

peak related to the amount of color contained in theobject. The output correlation intensity for each coloris

In(x',y'O0.) = Sf fb(xy)sb(x - f sinO - x',y - y')dxdyl2 , (2)

where fb(-) and Sb(-) are the binarized color encodedreference object and input object, respectively. It canbe seen from Eq. (2) that each output correlation func-tion is centered at x' = -f sinO, away from the opticalaxis. Thus, the proposed system can perform colorsignal detection by producing three different correla-tion signals at three different locations with the inten-sity of each correlation peak related to the spectralcontent of the object. For example, if a yellow inputobject (red and green) is used, the correlation functionintensity at the output plane can be written as

I(x',y') = I(x',y',01) + I2(xY',02), (3)

where the two correlation signals corresponding to thegreen and red color objects are diffracted at x' = -fsin 1 and x' = -f sinO2 away from the optical axis at theoutput plane.

It was noted previously that the relative correlationintensities at each location are related to the colorcontent of the object. For example, a color object withhigh red spectral content will produce a stronger out-put correlation signal at the red location comparedwith the correlation signals at the green and blue loca-tions. Thus, the pattern recognition system intro-duced here can detect the input object and determine

950 APPLIED OPTICS / Vol. 27, No. 5 / 1 March 1988

U....... ...U......@.. ....... 5...U .. ~~~~......

HEU......

,U.. . ............ ..

I::",.,.i^,..,3E,',',:j.......:..

.. a:::n~ru~u.:::r:::~.............E .. .... ...... ... i ....

Fig. 3. Black-and-white plot of the yellow input object.

its colors by analyzing the intensity of the correlationpeaks at three different correlation positions.

Ill. Computer Simulation

In this section we present a computer simulationanalysis of the real-time polychromatic binary correla-tor. For simplicity, we used a yellow character A as theinput color object; therefore, the matched filter is syn-thesized for two colors, red and green. The profile ofthe color input signal is plotted in Fig. 3. The systemdescribed here works well for both continuous toneobjects as well as binary objects. For continuous toneobjects, the spatially encoded input color object is firstthresholded and then color pattern recognition is per-formed. It has been shown previously1 0 that, for clas-sical matched filtering, the correlation signal obtainedby binarizing the continuous tone input object andusing the matched filter made from the binarized am-plitude input has superior performance compared withthe nonthresholded case. Thus, binarization of theinput object and use of the matched filter synthesizedfrom the thresholded input can improve the systemperformance in terms of the correlation SNR and peakamplitude.

The correlation studies are implemented using a 512X 512 point 2-D fast Fourier transform (FFT), and theresults are plotted using a 3-D plotting subroutine.The input color object is sampled by a red and greencolor grating and the result to be displayed on a MOdevice is shown in Fig. 4. The sampling period of thecolor grating for each color is four pixels. The diffract-ed Fourier spectra of the spatially sampled object isshown in Fig. 5. It is assumed that the color encodinggrating used in this analysis diffracts the red and greencontents of the input object horizontally and vertical-ly, respectively.

The submatched filter for red is synthesized by us-ing the corresponding diffracted spectra (first-orderhorizontal Fourier spectrum) modulated at a specificspatial carrier frequency and blocking the other or-ders. The submatched filter for green is synthesizedsimilarly by using the first-order vertical Fourier spec-trum modulated at a different spatial carrier frequen-cy. The diffracted spectra used for each color is shown

.. 1.S Im m. . .:.. 1 U... U1 1. .1 . . .d ..W. ...................

... " . a......a....

. .. ". X .~ mu , mm. m. u .

Fig. 4. Binarized color encoded coherent image.

0.67-

0.33-<

Fig. 5. Diffracted Fourier spectra of the yellow input object.

in Fig. 5. Figure 6 illustrates the correlation results forthe color character A. Figures 6(a), (b), and (c) corre-spond to the correlation intensity for green, red, andyellow character A, respectively. It is evident that thecorrelation signals due to the various colors are spatial-ly separated so that the spectral content of the objectcan be analyzed.

IV. Experiments

The experimental setup used to study the real-timepolychromatic binary correlator is shown in Fig. 1.The input color object illuminated by incoherent lightis imaged onto the CCD array using a 20-cm focallength achromatic imaging lens. The color encodinggrating used here consists of two superimposed grat-ings for green and red with the ruling in the horizontaland vetical directions, respectively.

The CCD is an IDETIX digital vision system camerawhich is interfaced to an IBM PC using an inter-face board. The spatially encoded color images arethresholded in the IBM PC. The threshold value isdetermined by evaluating the histogram of the pixelvalues and picking the median, The experiment usesa Semetex 48-by-48 element MO device. The binar-ized input data are transmitted over a serial link to theApple II computer. The MO device is controlled bythe Apple computer using the interface electronicsprovided by the manufacturer. Figure 7 illustratesthe block diagram of the color binary pattern recogni-tion system. -

The binarized spatially encoded images are read outusing a 20-mW He-Ne laser. A 100-cm focal lengthlens is used to obtain the Fourier transform of the

1 March 1988 / Vol. 27, No. 5 / APPLIED OPTICS 951

.............................. I

1.00

0.67

0.'33

0.00 .

1.00

0.67-

0.33I

0.0 -<

(a) (b) - (c)

Fig. 6. Correlation results for the color character A: (a) output correlation for a green A; (b) output correlation for a red color A; (c) outputcorrelation for a yellow color A.

Fig. 7. Block diagram of the polychromatic binary pattern recogni-tion system.

sampled images. The diffracted spectra of the binar-ized yellow character A is shown in Fig. 8. Thematched filter is recorded on a Kodak SO-253 type filmusing the 6328-A line. The submatched filters for redand green were made by using the first-order diffract-ed Fourier spectra modulated at 0 = 300 and 0 = 20°,respectively.

We have performed color correlation experimentsfor three different colored examples of character A asshown in Fig. 9. Figure 10 illustrates the correlationresults obtained by the real-time polychromatic binarycorrelator corresponding to the various colored inputobjects. As an example, Fig. 10(a) illustrates the corre-lation peak obtained with a red character A where thecorrelation peak is diffracted to the right-hand side ofthe photograph. Figure 10(b) is the correlation signalcorresponding to a green colored character A which isdiffracted to the left-hand side of the photograph.Finally, Figure 10(c) represents the correlation signalsfor a yellow character A which results in two discretecorrelation peaks at the output plane. it is evidentfrom these figures that the correlator introduced herecan perform binary pattern recognition of color im-ages.

V. Conclusion

We have described a real-time polychromatic binarypattern recognition system which can perform imagecorrelation based on both the spectral content andshape of the target. A color grating is used to spatiallyencode the input color images into red, green, and bluecoherent images. A binary SLM is employed to readout the spatially encoded images and a multiple spec-

Fig. 8. Diffracted Fourier spectra of the color encoded yellow char-acter A displayed on the MO device.

Fig. 9. Profile of the spatially encoded yellow character A.

tral-spatial matched filter is utilized to obtain thecorrelation signals. The results indicate that the sys-tem works well and has a high degree of color spatialselectivity. Experimental results and computer simu-lation analysis of the system are provided to study thesystem performance for different colored objects. Al-though, the MO device used in the experiments isbinary, it does not seriously degrade the performanceof the correlator and in some cases it can even improvethe correlation SNR. The proposed system can besuitable for many real-time practical applicationswhich require color sensitive pattern recognition.

952 APPLIED OPTICS / Vol. 27, No. 5 / 1 March 1988

(a)

(b)

(C)

Fig. 10. Correlation results obtained by the real-time polychromatic binarypattern recognition system: (a) output correlation corresponding to a red charac-,- AA- 1 .-,y .I_ z -! _ __ .. /. I I ..ter A; kD) output correlation corresponci

correlation corresponding

The MO device used in the experiments was loanedto us by Semetex Corp. We would like to thank MikeLane and Dave Cox for their technical assistance.This work was supported in part by the EngineeringFoundation and the Institute of Electrical & Electron-ics Engineers under grant RI-A-87-7.

References

1. A. B. VanderLugt, "Signal Detection by Complex Spatial Filter-ing," IEEE Trans. Inf. Theory IT-b0, 139 (1964).

2. W. P. Bleha et al., "Application of Liquid Crystal Light Valve toReal-Time Optical Data Processing," Opt. Eng. 17, 371 (1978).

3. H.-K. Liu, J. A. Davis, and R. A. Lilly, "Optical-Data-ProcessingProperties of a Liquid-Crystal Television Spatial Light Modula-tor," Opt. Lett. 10, 635 (1985).

4. W. D. Ross, D. Psaltis, and R. Anderson, "Two-DimensionalMagneto-Optic Spatial Light Modulator for Signal Processing,"

ng to a green character A; c) outputto a yellow character A.

Proc. Soc. Photo-Opt. Instrum. Eng. 341, 191 (1982). SignalProcessing V, 1982.

5. D. Psaltis, E. Paek, and S. Venkatesh, "Optical Image Correla-tion with a Binary Spatial Light Modulator," Opt. Eng. 23, 698(1984).

6. F. T. S. Yu and B. Javidi, "Experiments on Real-Time Polychro-matic Signal Detection by Matched Spatial Filtering, "Opt.Commun. 56, 384 (1986).

7. C. Warde, H. J. Caulfield, F. T. S. Yu, and J. E. Ludman, "Real-Time Joint Spectral-Spatial Matched Filtering," Opt. Commun.49, 241 (1984).

8. J. E. Ludman, B. Javidi, F. T. S. Yu, and C. Warde, "Real-TimeColor Pattern Recognition," Proc. Soc. Photo-Opt. nstrum.Eng. 465, 143 (1984).

9. F. T. S. Yu, S. Jutamulia, T. W. Lin, "Real-time polychromaticsignal detection using a color liquid crystal television," Opt.Eng., vol. 26, no. 5, 453 (1987).

10. J. L. Horner and H. 0. Bartelt, "Two-Bit Correlation," Appl.Opt. 24, 2889 (1985).

1 March 1988 / Vol. 27, No. 5 / APPLIED OPTICS 953

i