Machine vision system for surface inspection on brushed

industrial parts.

Nicolas Bonnota, Ralph Seulin, Frederic Merienne

Laboratoire Le2i, CNRS UMR 5158, University of Burgundy, Le Creusot, France.

ABSTRACT

This work aims at detecting defects on metallic industrial parts with streaked surface. The orientation of thoseparallel streaks is totally random. The searched defects are scratch and lack of machining. A specific machinevision system has been designed to deal with the particular inspected surface features. One image is acquiredwith an annular lighting in bright field and six images are acquired with a rotating lighting in dark field. Aparticular image processing is applied on the six images in order to get one image that represents all the revealedimperfections. A thresholding processing is then applied on this image in order to segment the imperfections. Atrained classification, created with well known typical objects of each class, is performed. The classification hasto recognize the different defects and the small imperfections that are not defects. The decision phase is used toknow if the defects are acceptable, and therefore if the inspected part is acceptable. Some acceptability rules aredefined for every defect class. The developed machine vision system has been implemented on an experimentalindustrial production line and it gives 2 % of sub-detection and 16 % of over-detection.

Keywords: Machine Vision, brushed metallic surface inspection, smart lighting, defect detection

1. INTRODUCTION

A manufacturer that produces metallic parts wants to automate its production line. All the steps of the produc-tion line are partially automated. Inspection is the last step that is completely performed by human operators.Several experts are visually searching defects. The human control implies an important subjectivity in the in-spection. Indeed the same part could be acceptable for an expert but not for another. The disparity betweenexperts is increased by the disparity, for the same expert, between the beginning and the end of the day. Besides,there is no reference for the relevance of the defects. No rules allow the experts to define exactly a part asacceptable or not.

The manufacturer wants an inspection system that reaches the human control quality. Many industrialactivities have benefited from machine vision systems1–5 and especially the metallic parts production. As humancontrol is visually performed, a machine vision system is a good way to automate the inspection. The experts’method used to detect the defects is to reveal the defects with a particular lighting. Therefore, this project willtake important benefits of using a machine vision system to inspect the parts.

Finally, the recorded results of the automated inspection would allow to observe exactly the evolution of thenumber and the type of detected defects.

1.1. Industrial part

The industrial part is metallic. Its surface is machined in order to obtain a flat and non smooth surface. Theresult of this machining is a streaked surface, where the streaks are parallel. We call it a streaked pattern (figure1). Those streaks are very important for the mechanical characteristics of the part. Because of the specificity ofthe machining, the orientation of the streaked pattern is totally random, and some secondary streaked patterns,with less intensity, could appear on the surface with another random orientation.

a Nicolas Bonnot, IUT, 12 rue de la fonderie, Le Creusot, FRANCE, - www.le2i.comE-mail: nbonnot@iutlecreusot.u-bourgogne.fr ; Phone: +33 (0)3.85.73.10.90 ; Fax: +33 (0)3.85.73.10.97

Figure 1. Streaked pattern

1.2. Defects

Two different defects can occur on the surface of a part. Scratch is due to the manipulation of the part after themachining and is viewed as a lengthened marking in the part. The position of scratches is completely random onthe part’s surface. Lack of machining is a region of the part without any streaks due to a flatness problem ofthe part which surface is not completely machined. A lack of machining is a rough region of the part’s surface.Lack of machining can appear on two different part’s zones. The first one is positioned on the edge of the part,the second is on its center. Figure 2 presents the typical defects.

a b c

Figure 2.

(a) Scratch(b) Lack of machining on the part’s edge.(c) Lack of machining on the part’s center.

2. ACQUISITION SETUP

2.1. Principle of revelation

2.1.1. Shape revelation

For each part, its position in the image can slightly vary. The position of the part in the image has to be wellknown in order to apply correctly the image processing on the part’s surface. A region of interest, representingthe inspected surface, has to be defined for each part. The part is imaged in bright field6 in order to reveal thesurface but not the environment of the part (figure 3). In bright field, the light is directly reflected by the surfacetowards the camera and the surface appears as white in the image. The streaked surface of the part is revealedin white, whereas the environment of the part is dark.

2.1.2. Defects revelation

The manufacturer wants the machine vision system to reach the quality of the human control. Its study hasshown that the defects are revealed when the streaked pattern is not visible. So the principle of the lighting isto reveal the defects but not the streaked pattern. The study of the interaction of the defects and the streakedpattern with light has shown that they are revealed when they are perpendicular to the direction of the lightand when they are imaged in dark field.6 In that case, the surface is bright in the image. When the directionof the light is parallel to the orientation of the streaked pattern (in dark field), the surface appears as dark inthe image (figure 4).

A scratch interacts with the light in the same way than a streak. When the light’s direction is perpendicularto the streaked pattern orientation, the part’s surface is imaged as white and the scratch is not visible. Thereforethe scratches are revealed when the light’s direction is parallel to the streaked pattern orientation (figure 5.a).Due to its rough aspect, a lack of machining diffuse the light in all the directions whatever the direction of theincident light. Therefore it is revealed with any light direction and is imaged with a medium intensity (figure5.b).

Figure 3. Revelation of the part’s position.

a b

Figure 4. Revelation of the streaked pattern.(a) Lighting perpendicular to the streaked pattern.(b) Lighting parallel to the streaked pattern.

a b

Figure 5.(a) Scratch revelation.(b) Lack of machining revelation.

Due to the random orientation of the streaked pattern a specific lighting device was designed and realized inorder to image the part with several orientations enabling an efficient revelation of the scratches.

2.2. Lighting and acquisition features

The design of the lighting device is the most important step of the project. Better the acquired images of thedefects are and simpler and stronger the image processing are.7, 8 A fixed camera images the inspected surfaceof the part. The camera images two parts. The camera sensor resolution is 768*580 pixels, coded upon 8 bits inblack and white. Seven images are acquired with two different lighting devices. One image is acquired with anannular lighting in bright field, in order to detect the exact position of the part. It is called ”annular image”. Siximages are then acquired with a rotating lighting in dark field, in order to reveal the scratches with different lightdirections and the lack of machining (figure 6.(a,b)). They are called ”rotating images”. This rotating lightingis composed of twelve pie segments that can be independently activated (figure 6.c). For each ”rotating image”,two opposite segments of 30◦ in , are lighting the industrial part. Therefore, when the six ”rotating images”are acquired, the industrial part receives light from all the directions. All the defects and streaked pattern thatexist on the surface are then revealed in the ”rotating images”. Figure 7 presents the seven images acquired fora scratch. Figure 7.a presents the ”annular image” and figure 7.(b,c,d,e,f,g) presents the six ”rotating images”.The corresponding lighting configuration is also schematically represented.

12

4

3

1122

44

3330°30°

1

23

4

30°30°

11

2233

44

30°

3

2

130°30°

33

22

11

Camera

Annular lighting

Rotating lighting

Part's support

1

2

4

3

Camera

Annular lighting

Rotating lighting

Part's support

11

22

44

33

a b c

Figure 6.

(a) Lighting device 3D view.(b) Lighting device front view.(c) Lighting device top view (12*30◦).

a b c d e f g

Figure 7.(a) Annular image

(b,c,d,e,f,g) Rotating images

3. IMAGE PROCESSING

The image processing are separated into three steps. The detection phase has to list the imperfections observedon the part. The classification phase has to recognize each imperfection. The decision phase has to decide if thepart is acceptable or not.

3.1. Detection phase

The detection phase is separated into three steps. In the first step the region of interest is created. In the secondstep the ”rotating images” that potentially reveal defects are selected. In the last step, a thresholding processingis performed in order to segment the imperfections.

3.1.1. Region of interest definition

In the first step of the detection phase, a region of interest, representing the inspected streaked surface, is created.A fixed threshold is performed on the ”annular image”, as the captured image is saturated (grey level = 255) inthe part region. In order to exclude the edge’s effects of the part, a morphological erosion processing, using a[7*7] structuring element, is applied on the mask.

a b c

Figure 8.

(a) Annular image(b) Thresholded annular image(c) Eroded image as the region of interest

3.1.2. ”Rotating image” selection

In the second step of the detection phase, the ”rotating images” are selected. A scratch is observed when thestreaked pattern is not revealed, that is when a ”rotating image” is dark. A lack of machining is revealed in each”rotating image”. Therefore we work only on the dark ”rotating images” that potentially reveal the defects. Thedark ”rotating images” are selected upon their mean and their standard deviation. A ”rotating image” with a lowmean and a low standard deviation means that the streaked pattern is not revealed (figure 9). The boundariesof mean and standard deviation (figure 10) were determined on a representative sample parts observation.

a b c

Mean (µ) 190 54 19Standard deviation (σ) 31 28 5

Figure 9.

(a) Streaks revealed : high µ , high or low σ

(b) Streaks revealed : low µ , high σ

(c) Streaks unrevealed : low µ , low σ

�����

�������

15 8

85 30

Mean (µ)

Standard Deviation (σ)

µ = -7.86σ +147.88

0

Figure 10. Image Selection’s rules

Defect revelation varies between the different ”rotating images”. Moreover, a defect can be partially revealedthrough different images; it is due to its orientation variation (figure 7.(d,e)). A specific image processing isapplied on the selected images in order to get one image that completely represents all the revealed imperfections.This image is called ”revealed image”. In a ”rotating image”, brighter a pixel, representing an imperfection, isand better the revelation is. The ”revealed image” (RI) is computed from the selected ”rotating images” (Roti)as follows:for each pixel P(x,y)

RIx,y = Max(Rotix,y) with i = [1, 6]end for

The ”rotating images” are so merged in a unique image where defects are fully revealed and well contrasted.

3.1.3. Thresholding processing

In the third step of the detection phase, a thresholding processing is applied on the ”revealed image” to detectthe imperfections. A segmented imperfection is called ”object” and do not necessarily refers to a defect. Twothresholds are computed on the ”revealed image”. The first fixed threshold, called ”shape threshold” (1), allowsus to segment the shape of the existing objects. After the shape thresholding is performed, an object can besegmented in different blobs. Therefore, a morphological closing processing is computed on the obtained imageafter the shape thresholding in order to connect the blobs of the object. An area filter is performed on theobjects in order to remove the small objects, which area is less than a fixed value determined from observations.A new region of interest (ROI), representing the detected objects, is created. The second adaptive threshold,called ”detection threshold” (2), is applied on the new region of interest. It allows the selection of the segmentedobjects that are bright enough to be potentially a defect (figure 11). The block diagram of the thresholdingprocessing performed on the ”revealed image” is presented in figure 12. For the ”rotating images” acquired infigure 7, the computed ”revealed image” is presented in figure 13.a, and the object’s detection is presented infigure 13.b.

The shape parameters and the intensity parameters of the objects are finally computed in order to performthe classification and the decision phases.

ShapeThreshold = Mean + CsteST

With CsteST = Fixed threshold value(1)

DetectionThreshold = Mean + CsteDT ∗ StandardDeviation

With CsteDT = V alue for the adaptive threshold(2)

3.2. Classification phase

The classification phase is based on a trained classification operator.9, 10 Due to the specific machining, manyobjects, that are not necessarily defects, are segmented on the surface. The classification has to recognize thoseimperfections in order to distinguish them from real defects. Moreover, there is not the same level of acceptabilitybetween the defect classes. The classification was created with well known typical objects of each class. Forthe classification’s training, the typical objects were all named by an expert. They are given as input to theclassifier, with their name, their shape and their intensity parameters. The classifier gives us some classificationrules that are applied to the detected objects. We obtain the name of every objects detected on the part.

����� �������

������ ������

�������� ������

��������������

��� ���������������

����� ��������� !" #$%$#&

'�()�����������*� �+,-�

.�)�/���),00 �������

.�)�/���1������2�� �+3��+�0�

Figure 11. Principles of the Shape and the Detection threshold.

Revealed Image

Shape Threshold

Detection Threshold

Connection of close objects ( Morphological Closing )

Small objects removal Object's Area < Minimum Area

Object’s ROI definition

Object’s shape parameter Object’s intensity parameter

Figure 12. Block diagram of the thresholding processing

a b

Figure 13.(a) Revealed image.(b) Detection image. Defects are surrounded in white.

3.3. Decision phase

The decision phase is the last step of the processing. It is used to know if defects are acceptable, and thereforeif the inspected part is acceptable. Some decision rules are defined for the defect class ”Scratch” and ”lackof Machining”. As no simple decision rules were found, a trained classification based on discriminant analysisis used in order to separate the acceptable defects from the non acceptable defects. These decision rules arecreated using the objects correctly recognized by the classification phase. The parts, on which these objectswere detected, were all inspected by an expert. The objects and the parts were defined as acceptable or nonacceptable. These definitions are compared with the results of the decision’s rules and allow the validation ofthese rules. The classes ”Acceptable” (Acc) and ”Non acceptable” (NAc) are defined for each defect class. Atthe end, a part will not be accepted if it contains at least one non acceptable defect.

4. RESULTS

The developed machine vision system was tested on 4000 parts. Each part was inspected by the machine visionsystem and the human operator. The comparison between the two controls are presented in table 1. Only theparts on which objects were detected, and so on classification and decision phase are performed, are presented.The reported percentages represent the correct decision (diagonal) and the confusion (out of diagonal).

SystemExpert NAc Acc

NAc 74.5% 25.5%Acc 23.5% 76.5%

Table 1. Global percentage of correct decision for the parts

Table 1 does not include the parts for which no objects were detected. Those parts are included in thecomputation of the over-detection and the sub-detection. The comparison between the global results of humanand machine control gives 2.1 % of sub-detection and 16.6 % of over-detection.

5. CONCLUSION AND FUTURE WORKS

The machine vision system is validated by the manufacturer. It has been implemented on an experimentalindustrial production line. Both human and machine control will be done on the parts in order to validate themachine vision system on a long time. The shape and intensity parameters of the non acceptable defects arestudied in order to observe the evolution of these defects. The characteristics and the number of the defects willgive information to the manufacturer if there is a problem in the production line. The machine vision systembrings important benefits to the manufacturer. The subjectivity induced by the human control is removed andenables an accurate monitoring of the factory’s production line.

The main restriction of this machine vision system is that it is not really adaptive. The major parametersvalues of the processing (image selection, threshold constants, ...) were determined from observations. Theresults presented in 4 are good for the defects and the surface’s aspect presented in 1.1 and 1.2. An importantevolution of the defects, or if a new defect occurs, it will induce very bad results. Moreover, the manufacturerdoes not exclude the possibility of modifying the aspect part’s surface. Therefore, future work concerns adaptiveprocessing in order to consider any future evolution.

REFERENCES

1. E. N. Malamasa, E. G. M. Petrakisa, M. Zervahisa, L. Petitb and J-D. Legat, “A survey on industrial visionsystems, applications and tools”, in Image and Vision Computing, Volume 21, pp. 171–1881, 2003.

2. T. Pfeifer and L. Wiegers, “Reliable tool wear monitoring by optimized image and illumination control inmachine vision”, in Measurement, Volume 28, Issue 3, pp. 209–218, 2000.

3. A. D. H. Thomas, M. G. Rodd, J. D. Holt and C. J Neill, “Real-time industrial Visual Inspection: A Review”,in Real-Time Imaging, Volume 1, Issue 2, pp. 139–158, June 1995.

4. AC. Legrand, E. Renier, P. Suzeau, F. Truchetet, P. Gorria and F. Meriaudeau, “Machine vision systems inmetallurgy industry”, in Journal of Electronic Imaging SPIE, 10(1), pp. 274–282, January 2001.

5. P. Bourgeat, F. Meriaudeau and P. Gorria, “Defect detection and classification on metallic part”, in Proc.SPIE Machine vision industrial Inspection X, San-Jose, USA, Volume 4664, pp. 182–189, January 2002.

6. F. Pernkopf and P. O’Leary, “Image acquisition techniques for automatic visual inspection of metallic sur-faces”, in NDT & E International, Volume 36, Issue 8, pp. 609–617,December 2003.

7. R. Seulin, N. Bonnot, F. Merienne, P. Gorria, “Simulation process for the design and optimization of a ma-chine vision system for specular surface inspection”, in Conference on Machine Vision and Three-DimensionalImaging Systems for Inspection and Metrology II, SPIE, Boston, USA, 4567, pp. 129–140, October 2001.

8. R. Seulin, F. Merienne, P. Gorria, “Simulation of specular surface imaging based on computer graphics :application on a vision inspection system”, in Journal of Applied Signal Processing - Special issue on AppliedVisual Inspection, EURASIP, 2002 (7), pp. 649–658, July 2002.

9. P. Bourgeat, K. Tobin, F. Meriaudeau, P. Gorria, “Pattern Wafer segmentation”, in Journal of the Societyof Manufacturing Engineers, Volume 242, pp. 2–11, January 2003.

10. J. Miteran, S. Bouillant and E. Bourennane “SVM approximation for real-time image segmentation by usingan improved hyperrectangles-based method”, in Real-Time Imaging, Volume 9, Issue 3, pp. 179–188, June2003.