International Journal of Production Research An effective and efficient …islab.kaist.ac.kr/chungcw/tft-lcd.pdf · 2015. 11. 26. · International Journal of Production Research

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [2007-2008-2009 HanKuk University of Foreign Studies]On: 12 July 2010Access details: Access Details: [subscription number 907464914]Publisher Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Production ResearchPublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t713696255

An effective and efficient defect inspection system for TFT-LCD polarisedfilms using adaptive thresholds and shape-based image analysesChung-Ho Noha; Seok-Lyong Leea; Deok-Hwan Kimb; Chin-Wan Chungc; Sang-Hee Kimd

a School of Industrial and Management Engineering, Hankuk University of Foreign Studies, Yongin-shi, Korea b School of Electronics Engineering, Inha University, Yonghyun-dong, Incheon-shi, Korea c

Division of Computer Science, KAIST, Daejeon-shi, Korea d Key Technology Research Center, Agencyfor Defense Development, Daejeon-shi, Korea

First published on: 21 August 2009

To cite this Article Noh, Chung-Ho , Lee, Seok-Lyong , Kim, Deok-Hwan , Chung, Chin-Wan and Kim, Sang-Hee(2010)'An effective and efficient defect inspection system for TFT-LCD polarised films using adaptive thresholds and shape-based image analyses', International Journal of Production Research, 48: 17, 5115 — 5135, First published on: 21 August2009 (iFirst)To link to this Article: DOI: 10.1080/00207540903117899URL: http://dx.doi.org/10.1080/00207540903117899

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

http://www.informaworld.com/smpp/title~content=t713696255

http://dx.doi.org/10.1080/00207540903117899

http://www.informaworld.com/terms-and-conditions-of-access.pdf

International Journal of Production ResearchVol. 48, No. 17, 1 September 2010, 5115–5135

An effective and efficient defect inspection system for TFT-LCD

polarised films using adaptive thresholds and shape-based image analyses

Chung-Ho Noha, Seok-Lyong Leea*, Deok-Hwan Kimb, Chin-Wan Chungc andSang-Hee Kimd

aSchool of Industrial and Management Engineering, Hankuk University of Foreign Studies,89 Mohyun-myon, Yongin-shi, Korea; bSchool of Electronics Engineering, Inha University,Yonghyun-dong, Incheon-shi, Korea; cDivision of Computer Science, KAIST, Daejeon-shi,

Korea; dKey Technology Research Center, Agency for Defense Development, Daejeon-shi, Korea

(Received 19 November 2008; final version received 6 June 2009)

Defect inspection of polarised films is a crucial process in TFT-LCD (thin filmtransistor–liquid crystal display) production. In this paper we propose a defectinspection system for TFT-LCD film images that detects film defects in a real-time production environment and classifies them based on the type of a defect.The proposed system is designed to locate defects promptly using an adaptivethreshold technique and determines the defect type through image analysis usingvarious features, such as the geometric characteristics and the shape descriptorwith intensity distribution called the shapeþID descriptor. The experimental resultsusing a set of test images obtained in a real production line are quite promising.Compared with existing methods, our method identifies defects effectivelyand efficiently enough to be used in a real-time environment. It also achievesa high correctness in classifying the defect type for various types of defects.In addition, it demonstrates robustness with respect to scale and rotationaltransformation.

Keywords: automated inspection; computer vision; data based management; datamining; information systems; information technology; product development

1. Introduction

Display devices provide visual interfaces to humans by providing visual informationtransmitted from electronic devices, such as televisions, computers, digital watches,various measuring instruments, and mobile devices. TFT-LCD is a thin and flat devicethat is composed of colour or monochrome pixels arrayed in the front of a light source,and has become far superior to other types of display devices, such as CRT (cathode raytube), PDP (plasma display panel), and FED (field emission display). The use of TFT-LCD has increased rapidly since 1990. Currently, it has more than an 80% market share ofthe flat display panel market. There is keen competition in the TFT-LCD industry, andmany companies are making desperate efforts to increase their market share. Therefore, itis important to produce high-quality products and enhance productivity in themanufacturing process.

*Corresponding author. Email: [email protected]

ISSN 0020–7543 print/ISSN 1366–588X online

� 2010 Taylor & Francis

DOI: 10.1080/00207540903117899

http://www.informaworld.com

Downloaded By: [2007-2008-2009 HanKuk University of Foreign Studies] At: 06:12 12 July 2010

A polarised film is a critical element of TFT-LCD. Various defects might be

generated during the process of film attachment when manufacturing TFT-LCD. An

examination with the naked human eye is not suitable because the film moves fast along

the production line and some defects are too small to be recognised by the human eye.

In order to detect these defects, it is essential to use an automated defect inspection

system, which is a device used in the final step of the film manufacturing process that

extracts defect regions automatically and sends recognition signals to a central

controlling unit (Yoon et al. 2008). The system plays an important role during TFT-

LCD production since it prevents possible malfunctions by detecting defects in a timely

manner and reduces the time needed to identify inferior products. In this paper we

present the design and implementation of an automated defect inspection system for

polarised film images of TFT-LCD. The system first detects film defects promptly using

an adaptive threshold technique since it is used in a real-time environment without

stopping the production line (real-time mode). Then, it classifies the defect types through

defect image analysis using various features such as the geometric characteristics and

shape descriptor with the intensity distribution called a shapeþID descriptor (non real-time

mode).A defect inspection system is an application of a machine vision system that generally

consists of three subsystems: film image acquisition, defect inspection, and marking and

presentation. There are various types of defects, and the configuration of cameras and

illumination devices varies according to the type of defects to be embossed. A specific type

of defect requires a specific configuration. The defect film images used in this paper are

acquired using line scan cameras with transmitted light, and include different types of

defects appearing in the released film. Using the inspection system, engineers can obtain

information related to the problems that occur, and cope with various defects that

frequently occur in the TFT-LCD manufacturing process in a timely manner. They can

also enhance the product quality and productivity by modulating production environment

variables, such as the camera setup, illumination, temperature, and humidity, depending

on the inspection results.Our inspection system is implemented to handle six types of defects: dents, scratches,

black and white spots, swollen defects, and craters, which appear frequently and have a

significant influence on the quality of the final product. Dents are generated when

adhesion layers are pressed down by adhesives or other alien substances. Scratches are

defects with a thin and long shape and are produced when the surfaces of films are

scratched. Black and white spots have a small point shape and appear frequently as dust

and other materials. Swollen defects occur when film layers are inflated by gas, and craters

are clear and large with a volcanic crater shape. Figure 1 shows the defect images captured

by CCD (charge coupled devices) line-scan cameras with a 7K 100MHz data rate and

LED illumination sources.We do not address image acquisition and defect marking since they are beyond the

scope of this paper. Instead, we focus on an image processing unit to find and classify

defects effectively and efficiently using image analysis algorithms. A variety of methods

have been developed for defect inspection, as shown in Section 2. However, most of

them address the detection of defects without considering their classification. Although

they consider the classification, they focus on categorising only a few types of defects.

In addition, most methods do not consider real-time constraints in locating defects that

should be considered important in a production environment. Considering the

5116 C.-H. Noh et al.


weakness of existing methods, the contributions of our work are summarised asfollows.

. Our system detects film defects efficiently in a real-time environment with higheffectiveness (precision¼ 1.00, recall¼ 0.99) without stopping the production line.Therefore, it is very usable in a real production field in the TFT-LCD industry.

. Our classification algorithm demonstrates considerable correctness (preci-sion¼ 0.78, recall¼ 0.78) in determining the defect types, which leads to animprovement in the quality of the final product and an enhancement ofproductivity during the manufacturing process.

. Our inspection system is capable of classifying various defects (up to six types ofdefects). To our knowledge, there are few methods or systems that can supportmore than five types of defects in a single configuration.

. Our shapeþID descriptor used to determine large-sized defects can be applied toother potential applications such as image retrieval, topographical search, objectrecognition, and medical image analysis.

The remainder of the paper is organised as follows. Section 2 provides a survey ofrelated work with a discussion of defect inspection and image analysis methods. In Section3 we present the proposed study including defect detection using an adaptive thresholdtechnique, and defect classification using image analysis methods. Section 4 providesexperimental results and analyses on the proposed system, and finally we give conclusionsand future research plans in Section 5.

2. Related work

Due to the considerable competition in the flat display panel industry over the last decade,automated defect inspection has been studied in many organisations, such as TFT-LCD

(c) Black spot

(f) Crater

(a) Dent (b) Scratch

(d) White spots (e) Swollen defect

Figure 1. Defect images captured by line-scan cameras and LED illumination sources.

International Journal of Production Research 5117


manufacturing companies, laboratories, and research institutes. Although automatedinspection has become increasingly important, only a few studies have been openlypublished for security reasons. In reality, most companies regard the information relatedto these works as confidential, and are unwilling to release it. Lu and Tsai (2004) proposeda global approach for an automatic visual inspection of micro-defects in patterned TFT-LCD surfaces. Their method was based on an image reconstruction scheme using singularvalue decomposition (SVD), and accomplished considerable effectiveness in identifyingdefects. However, it has restrictions when being applied in real-time detection due to theconsiderable overheads incurred in decomposing a defect image into eigenvalue–eigenvector factorisations and reconstructing the image.

There are a few studies that use an adaptive threshold technique. Kim et al. (2004)introduced an automated inspection method using the statistical characteristics of localblocks and pattern elimination techniques based on the pixel difference and adaptivemultilevel threshold technique. The method is fast but it has a weakness in defectidentification. The standard deviation becomes large if a defect is large or there are severaldefects distributed over an image, making precise segmentation difficult and resulting inpoor defect identification. Oh et al. (2004) used a directional filter bank (DFB) and anadaptive multilevel threshold technique to find line-type defects in TFT-LCD panels. TheDFB was used to identify a line-shape abnormal region of a low-resolution image, while amultilevel threshold technique was used to detect abnormal line defects of a high-resolution image. However, the method has a weakness in that it identifies only line-typedefects, preventing it from being used in diverse applications.

There are many studies focusing on determining a specific defect, known as mura,which is a non-uniformity defect. Jiang et al. (2005) attempted to develop a more objectivemethod to determine mura defects in LCD panels. This method consists of two phases: onefor carrying out an analysis of the variance to detect the existence of defects, and the otherfor determining the location and size of the defects. However, the method focuses ondetection only, without considering any classification. Lee and Yoo (2004) proposed anautomatic inspection method that detected and quantified TFT-LCD region-mura defects.They first segmented candidate regions from TFT-LCD panel images using modifiedregression diagnostics and thresholds, and then quantified the mura level for eachcandidate. The method detects spot-type defects well, but does not work properly for othertypes of defects. Baek et al. (2004) proposed an automated inspection algorithm using apolynomial approximation as well as an optimal threshold technique to obtain anobjective decision level for judging defects. Zhang and Zhang (2005) also introduced afuzzy logic and neural network approach to quantify mura defects to cope with thecomplexity and vagueness of the defects. However, it is difficult to obtain a polynomialapproximation and quantify defects using these two methods, which prevents them frombeing applied in a real-time environment. In order to classify multiple types of defects,Yoon et al. (2008) proposed a defect inspection system for polarised film images, whichutilised region-growing based image segmentation using GLCM (gray level co-occurrencematrix) and template matching techniques. They classified micro-defects that needed to bemagnified before processing. This degrades the runtime efficiency, making a real-timeprocess difficult.

There have been attempts to transform film images into a frequency domain andanalyse the features in the frequency domain since the transformation makes it possible toprocess an image effectively for locating defects. Tsai and Hung (2005) addressed theproblem of automatic inspection of micro-defects including pinholes, scratches and



particles in TFT-LCD panel surfaces using a global one-dimensional (1D) Fourier-basedreconstruction scheme. By improving this scheme, Tsai et al. (2007) proposed a non-referential detection scheme that worked directly on the 1D line images using a Fourierimage reconstruction. The Fourier reconstruction process can remove the complicatedbackground pattern and preserve local anomalies properly. However, there are restrictionsin detecting defects using the Fourier transform when the frequency component of a defectmay not have a repeated pattern.

On the other hand, various features, such as intensity, texture map, colour histograms,and shape descriptors, are used to represent defects. Indeed, there are a variety of methodsin the image processing domain to describe defect regions. To determine the type of defect,it is usual to use a shape feature since the type is well described by the shape of the defect.One well-known method is the shape context (Belongie and Malik 2000, Mori et al. 2005),which represents the shape of objects, measures the shape similarity, and recovers the pointcorrespondence. It describes the coarse arrangement of a shape with respect to a pointinside or on the boundary of the shape. The shape context is represented as a vector-valuedattribute in a bipartite graph matching that can be used for object recognition andsimilarity-based querying. We have made some modifications to the existing shape contextto accommodate the requirements for determining the type of film defect, and haveobtained more discriminating power.

Most methods mentioned above focus on determining the existence of defects withoutconsidering the classification of the defects. Otherwise, they support classifying only a fewspecific types of defects. In addition, they do not consider the real-time constraints, whichare important in a production environment. Some methods have developed techniques foreliminating regular patterns that exist in the background of TFT-LCD panel images andcompensating for the irregularity of film images, which results from non-uniformillumination. The current growth of optical technology such as LED illumination sourcesand precision cameras makes it possible to acquire uniform and high-quality film images,and therefore the irregularity problems of films have been solved to a large degree. Animportant consideration in manufacturing TFT-LCD is to locate defects promptly in areal-time environment, and to determine the type of defect effectively for productivity andproduct quality.

3. The proposed defect inspection system

A film defect inspection system consists of the following three subsystems: (1) an imageacquisition subsystem that obtains analog data from pairs of cameras and illuminationsources, and transforms them into digital images using grabber boards and computersoftware; (2) a defect inspection subsystem that detects and classifies film defects; and (3) amarking and presentation subsystem that places a mark on the defect region, and displaysand prints it on an appropriate output device. Figure 2 shows the entire setup of the filmdefect inspection system, including image data acquisition, detection and classification ofdefects, marking, displaying, and reporting of defects. A line of four cameras is used toacquire an image at different locations to detect different types of defects. In this study, aset of CCD line-scan cameras with a 7K 100MHz data rate were used. The sources of lightwere 7.3W Bar-style White LEDs and a 24V 1.8A controller with 100KHz luminousfrequency. The film images obtained by the data acquisition subsystem are analysed by adefect inspection subsystem in a control rack to locate defects and classify them.



When defects are detected in the film image, a signal is transmitted to the marking andpresentation subsystem to indicate the location where they are found and to display andprint them on output devices. An encoding system is used to measure the revolution speed

of rollers and provide a marking unit with the location information of defects. In thispaper we focus on developing a defect inspection subsystem since other subsystems arebeyond the scope of this study. The major tasks are first to detect film defects and then to

determine the types of film defects.

3.1 Detecting film defects

The characteristics of TFT-LCD film images vary according to the influence of the

production environment, such as illumination, camera setup, temperature, and manu-facturing process. Film images are divided into two categories according to the coarsenessof the background: fine-background and coarse-background images, where the back-

ground indicates a defect-free region. To detect defects, it is important to consider thecharacteristics of the background. Figure 3 shows two defect images with fine and coarsebackgrounds and their 2D intensity histograms, where a different colour is used for pixelswith intensity higher than a given threshold. Clearly, a coarse-background image has a

wider intensity distribution than a fine-background image.The intensity distribution of defect images has different characteristics from a normal

image without defects. The intensity of the normal image fluctuates within a narrow range,while that of a defect image varies widely. Based on that notion, we are able to draw thevariation of the standard deviations along the horizontal and vertical axes of the 2D image

with and without a defect, as shown in Figures 4(a) and (b), respectively. In the case of adefect image, the distribution of standard deviations shows large fluctuations in the regionof the defect, while there is little variation in a defect-free image.

CameraIllumination

Illumination

Camera MarkingUnit

Encoder

Encoder Distributor

Control Rack

InspectionUnit

Data controllingUnit

RS 232C

DisplayDeviceUser Server

Roller

Film

Figure 2. The entire setup of a film defect inspection system.



From the above observations, if the distribution of standard deviations for a set ofpixels is within a specific range (threshold), it can be safely assumed that the set of pixelsbelongs to a background. If the distribution of a specific set of pixels is beyond the range,they are regarded as candidate pixels of a defect region. This property is used to identifydefects. First, the standard deviations were lined up in ascending order with respect to eachaxis. Figure 5 shows the distribution of the standard deviations for a defect image withrespect to the horizontal axis and its lining-up in ascending order.

1

12 23 34 45 56 67 78

S1

S20

S39

S58

S77

50

100

150

200

250

1

13 25 37 49 61 73 85

S1

S22

S43

S64

S85

50

100

150

200

250

(a) A defect image with fine background

(c) A 2D intensity histogram for (a) (d) A 2D intensity histogram for (b)

(b) A defect image with coarse background

Figure 3. Defect images with fine and coarse backgrounds and their 2D histograms.

(a) A film image with defect (b) A film image without defect

Figure 4. Change in the standard deviations along the horizontal and vertical axes of a 2D image.



As shown in Figure 5(b), there is a sudden change at the 78th line, which indicates the

boundary between a defect and the background. The set of pixels up to the boundary (� of

all pixels) is regarded as the background. The parameter � was determined experimentally.

The mean (�b) and standard deviation (�b) of the background can then be calculated.

Using the control chart, a well-known statistical analysis technique, each pixel is examined

to determine if it belongs to a defect region. The threshold, �b� k�b, is determined

adaptively since �b and �b vary according to the film image

IðiÞ isa background pixel, when�b � k�b � IðiÞ � �b þ k�b,a defect pixel, otherwise,

�ð1Þ

where I(i) is the intensity of the ith pixel and k is the parameter used to compute the

threshold for the defect region. A more accurate threshold can be obtained by considering

defect regions and backgrounds separately using the distribution of standard deviations

instead of the pixel intensity itself. If defect regions are included in computing �b, a larger

value may be obtained due to the low uniformity of the defect regions, degrading the local

adaptiveness in determining the threshold. The precision becomes lower if a larger k-value

is used to avoid this. Figure 6 shows the segmentation result when the pixel intensity and

distribution of standard deviations are considered. A better result can be obtained when

the latter is used.After all pixels have been examined, the defect region is determined using the labeling

technique, which merges the defect pixels to produce candidate defect regions considering

Figure 5. Distribution of the standard deviations and its lining-up.

(a) Original images (c) Using the distribution of sb(b) Using the pixel intensity

Figure 6. Segmentation result using the pixel intensity and distribution of standard deviations.



the spatial relationship between them and eliminates noise. A region with fewer defect

pixels than the specified number is considered to be noise. Figure 7 shows the defect images

presented in Figure 1 and the corresponding binary images with defect regions.

3.2 Classifying film defects

After identifying the defects, they are classified so that each defect can be assigned to a

specific type. Unfortunately, there is no standardised rule or regulation to classify and

name defects. In this study, six types of defects that occur frequently in a TFT-LCD

production environment were selected and named by referring to the related literature and

industry. In reality, there are a wide variety of defects at the production site and new types

are being introduced as new equipment is developed. The defects not included in the above

six types are classified as an unknown type. This type can be handled manually, but it is

important to discriminate the type for post-processing. The classification process is

performed in such a way that the types that are discriminated easily are examined first and

the more difficult ones are examined next. Some defects have different sizes, such as dents

and swollen defects. These are separated into two categories based on the defect size, small

(s-) and large (l-) defects, since different methods need to be applied for effective

classification. In this paper, scratches are first examined because they are thin and long-

shaped and can be classified easily. Next, small-sized defects, such as black and white

spots, s-dents, and s-swollen defects are examined. Large-sized defects, such as craters,

l-dents and l-swollen defects are examined last.

3.2.1 Classification of scratches

The shape of scratches is similar to a straight line and is easy to discriminate. Simple

geometric features are used to distinguish them from other features such as colour and

texture. An MBR (minimum bounding rectangle) is defined as the minimum rectangle that

precisely encloses a defect. In Figure 8, the MBRs of two different scratches are depicted

by rectangles. A white-coloured region within each MBR indicates a defect itself, and is

defined as an actual defect region (ADR). Using MBR and ADR, two geometric features

Figure 7. Defect images and the corresponding binary images with defect regions.



can be drawn as follows:

thin ratio ¼minðwidth of MBR, height of MBRÞ

maxðwidth of MBR, height of MBRÞ, ð2Þ

adr ratio ¼#pixels in ADR

#pixels in MBR, ð3Þ

where thin_ratio indicates the thinness of the MBR, and adr_ratio represents the ratio ofthe area between MBR and ADR. These two values are used to identify a scratch, andtheir thresholds are determined experimentally. In this paper, if the thin_ratio OR theadr_ratio of a defect is less than 0.4, then it is classified as a scratch defect.

3.2.2 Classification of small-sized defects

If the size of a defect is small, it is difficult to recognise and classify, since insufficientfeatures can be extracted to distinguish it from other defects. There are four types of small-sized defects: black and white spots, s-dents, and s-swollen defects. These defects have asimilar shape (circle or ellipse) and thus it is better to use the composition of white/blackcolours than geometric or shape-based features. The defect region is first divided into twosub-regions (white and black regions). As a criterion to determine if a pixel is white/black,the mean (�b) of the background, which is used in Section 3.1, was used. If the intensityvalue of the pixel is less than �b, it is black, otherwise it is white. Based on this notion, theblack ratio of a defect region can be calculated as follows:

black ratio ¼#fpj 2DRjIð pjÞ5�bg

#fpj 2DRg, ð4Þ

where DR is the set of pixels in the defect region, pj is the jth pixel in DR, and I(pj) is theintensity of pj. Figure 9 shows the composition of white/black colours for the four types ofdefects. As shown in the figure, the black and white spots are clearly distinguished,whereas s-dents and s-swollen defects are not. If the black_ratio of a defect is less than thethreshold wt, it is classified as a white spot. On the other hand, it is classified as a blackspot if the black_ratio is larger than the threshold bt. In the case of s-dents and s-swollendefects, another method is needed to determine the type. As shown in Figures 7(a) and (e),the black and white sub-regions of s-dents and s-swollen defects are always placed in

Figure 8. Minimum bounding rectangles for scratch defects.



reverse order since the location of cameras and the direction of the light in a specificproduction line are fixed. Using these characteristics, a defect with a white sub-region inthe upper position and one in the lower position can be considered as an s-dent ands-swollen defect, respectively.

3.2.3 Classification of large-sized defects

For relatively large-sized defects it is effective to use a shape-based feature since the shapesof the defects are distinguishable from each other. A shape matching method using theshape context (Belongie and Malik 2000, Mori et al. 2005) is one of several methods usedto measure the similarity between shapes. It employs a rich local descriptor, and isdifferent from other shape matching methods, which utilise the brightness at a single pixelor edge location. It maintains a log-polar histogram with K bins, each of which holds thenumber of points, and uses the �2 distance when measuring the similarity between points.In this paper we suggest a new method that adds the intensity distribution over film defectsto an existing shape context to effectively classify the type of defect. The next sectionbriefly describes the shape context method to make the paper self-contained.

3.2.3.1 The shape context. In the shape context method (Belongie and Malik 2000, Moriet al. 2005), a shape is represented by a discrete set of points that are sampled from theinternal or external contours of the shape. These can be obtained by the locations of edgepixels, giving us a set P¼ {p1, . . . , pn}, pi 2R

2 , of n points. Consider the set of vectorsoriginating from a point to all other sample points on the shape. These n� 1 vectorsexpress the configuration of the entire shape relative to the reference point. For a point pion the shape, a histogram hi of the relative coordinates of the remaining n� 1 points isdefined as the shape context of pi and is expressed as follows:

hiðkÞ ¼ #fq 6¼ pi : ðq� pi 2 binðkÞÞg, ð5Þ

where bin(k) is the kth bin of hi. Consider the insect image in Figure 10(a). Random samplepoints were chosen from the edge points as shown in Figure 10(b), and the vectors

0

0.2

0.4

0.6

0.8

1

1.2

Figure 9. The composition of the white/black colours for the four types of defects.



originating from a point to all other points in the shape is depicted in Figure 10(c). Themedian distance � for all N2 point pairs is shown at the bottom for reference. Thisproduces a compact descriptor for each sample point by computing a coarse histogram ofthe relative coordinates of the remaining points. The reference orientation for thecoordinate system can be absolute or relative to a given axis and this choice depends on theproblem setting. Figure 10(d) shows a log-polar coordinate system with 5� 12 bins. Figure10(e) gives an example of the log r, � histogram computed for the set of vectors inFigure 10(c), where the dark bins denote larger values.

In determining the shape correspondence, the matching cost is computed which iscomprised of two terms: one for the shape context and the other for local appearance. Thelatter is not mentioned since it is not related to our method. The shape context term Cs isgiven by the �2 distance between the two histograms. If the two histograms are denoted byhi(k) and hj(k), respectively, Cs can be computed as follows:

Cs ¼1

2

XKk¼1

½hiðkÞ � hj ðkÞ�2

hiðkÞ þ hj ðkÞ: ð6Þ

3.2.3.2 The shapeþID descriptor. A shape is usually represented by two classes ofdescriptors, i.e. contour-based and region-based descriptors. The shape context is basicallya contour-based method that quantises points at the contour of an object. To classify film

Figure 10. Log-polar histogram of the shape context.



defects effectively, it is important to consider the region-based descriptor since there aremany changes to the region inside film defects. In this section we introduce a new shapefeature descriptor that combines an existing shape context with an intensity distributionthat describes the change in pixel intensity inside a film defect. Hereafter, we call it theshapeþID descriptor, where ID means ‘intensity distribution’. To compute the intensitydistribution, the log polar histogram, which is similar to that of the shape context, ismaintained, but the information on the intensity difference between background pixels andpixels included in a specific bin is also added.

The shapeþID descriptor is generated using the following procedure. First, all the edgepoints that surround defects are extracted using a Canny edge detector (Michael et al.1997), which is a well-known detection method. Next, a fixed number of edge points aresampled. The following simple technique is used to obtain the sample points that aredistributed as uniformly as possible: after a point is selected randomly from a set of edgepoints, the nearest point to the selected point is excluded from the set. An almostuniform distribution of sample points can be obtained using this simple technique.Finally, a log polar histogram is calculated with respect to each sample point, whereeach bin contains the number of points and intensity difference between backgroundpixels and pixels in the bin. Consider a log polar histogram with the intensitydistribution shown in Figure 11. The kth bin, bin(k), of the histogram with respect to apoint pi has two components, shi(k) and ihi(k), that hold the number of points, as shownin the shape context, and the difference in intensity between the background pixels andpixels in the bin, respectively.

The shapeþID descriptor is represented by shi(k) and ihi(k), where the values of shi(k)and ihi(k) are normalised to be in the range [0, 1] for computational convenience. Therationale for using the intensity difference is that the types of defects are oftendiscriminated by the difference in intensity from the background, since the intensity isalmost invariable in the same production line. This is because the camera setting,illumination, and the film moving speed are normally consistent in the line. The shi(k) and

Figure 11. A log polar histogram with the intensity distribution.



ihi(k) values are computed as follows:

shiðkÞ ¼ #fq 6¼ pi : ðq� pi 2 binðkÞÞg, ð7Þ

ihiðkÞ ¼ �ik � �b, where �ik ¼

PIðl Þ

#ðl Þ, l2PT, l� pi 2 binðkÞ: ð8Þ

In this equation, I(l) is the intensity of pixel l, PT is the set of entire pixels in the image,�ik is the normalised mean of the intensity of the pixels of the kth bin with respect to theith edge point, and �b is the normalised mean of the intensity of background pixels.Considering the intensity distribution with the shape context, defects can be classifiedmore effectively due to the stronger discriminating power, particularly for those defectswith similar contours, such as dents and swollen defects. In addition, it provides a morerobust classification when there are diverse environment variables, such as the illuminationdifference over defect images, since it considers the intensity difference with respect to thebackground instead of the intensity itself. Figure 11 gives two examples of computing ihi(k)using the above expression.

3.2.3.3 Determining the type of defect. The matching cost between two film defect imagesis calculated to determine the type of a large-sized defect. The matching cost consists oftwo terms calculated using the shapeþID descriptor: one for shape dissimilarity and theother for intensity distribution dissimilarity. The shape dissimilarity is given by thecost CS, which is the distance between two histograms (sh(k)), while the intensitydistribution dissimilarity is represented by the cost CI, which is the difference between twohistograms (ih(k)).

Consider a point pi on the first defect image and a point pj on the second defect image.To calculate the cost using the shapeþID descriptor, two K-bin histograms, sh(k) and ih(k),are considered separately. The cost CS(i, j) for sh(k) and the cost CI(i, j) for ih(k) between thetwo points pi and pj are calculated as follows:

CSði, j Þ ¼1

2

XKk¼1

½shiðkÞ � shj ðkÞ�2

shiðkÞ þ shj ðkÞ, ð9Þ

CIði, j Þ ¼1

2

XKk¼1

½ihiðkÞ � ihj ðkÞ�2

jihiðkÞj þ jihj ðkÞj: ð10Þ

Considering the two components of cost, the combined cost C(i,j) between two points piand pj is defined as C(i,j)¼ (1�!)CS(i,j)þ!CI(i,j)(0 � !� 1), where ! is the weight. Theweight is determined based on the user’s preference or the relative importance between theshape feature and the intensity distribution. In our experiment they were assumed to haveequal importance, i.e. !¼ 0.5. Given the set of costs C(i,j) between all pairs of points pi onthe first image and pj on the second image, it is important to minimise the total matchingcost. This is an example of the weighted bipartite matching problem, which can be solvedin O(n3) time (Belongie and Malik 2000, Mori et al. 2005). In this paper we use the moreefficient algorithm (Joncker and Volgenant 1987). The input to the algorithm is a squarecost matrix with entries C(i,j). The result is a permutation �(i) such that the sum

Pi Cði,�ðiÞÞ

is a minimum.The template matching method is used to determine the type of defect. A template

database stores feature values for referential defect images, the types of which have been



classified in advance. A set of representative defect images is chosen for each defect type

according to domain experts’ recommendation and stored in the template database. When

a defect image is examined, the method first produces the shapeþID descriptor of a given

defect image, and then calculates the cost difference between the defect image and

referential defect images in the database. The type of given defect is determined to be the

same as that of the best matching (i.e. with a minimum cost) defect image among the

referential defect images. If the given image is not matched to any referential defect image

in the database (i.e. the minimum cost is larger than a specified threshold TH), it is

regarded as an unknown type. Large-sized defects, such as craters, l-dents, and l-swollen

defects, are classified using the matching cost function.

4. Empirical study

4.1 Preparation for experiments

To evaluate the effectiveness and efficiency of the proposed method, we have implemented

a film defect inspection system using Microsoft Visual Cþþ. The experiment was run on a

Xeon 2.5GHz Dual CPU. The inspection system is placed at the defect inspection

subsystem of the control rack in Figure 2, and designed to locate defects from film images

and classify them according to the type of defect. Experimental film image data sets were

acquired from real production lines, and are composed of 406 film images including 218

images with defects and 188 defect-free images. The data sets are separated into two sub-

sets, i.e. training sets for parameter optimisation and validation sets for evaluating the

proposed method using the parameters determined by the training sets. The training sets

include 135 film images while the validation sets include 271 images. Tables 1 and 2 show

the structure of the experimental data sets and the five sample defect images for each type,

respectively.The precision and recall, which are well known in information retrieval and

classification applications, were used to evaluate the effectiveness of our method. Let

Set(pred) be a set of predicted objects classified by our method, and Set(act) be a set of

actual objects classified by domain experts. When the number of elements in Set is denoted

by jSetj, the precision and recall are defined as follows:

precision ¼Set ð pred Þ \ Set ðactÞ��

Set ð pred Þ�� , recall ¼

Set ð pred Þ \ Set ðactÞ��

Set ðactÞ�� : ð11Þ

Table 1. Structure of the experimental film image data sets.

Data set

Defect type

Defect-free TotalDent ScratchBlackspot

Whitespot

Swollendefect Crater Un-known

Training set 9 13 10 16 14 8 2 63 135Validation set 19 25 20 32 29 17 4 125 271



4.2 Detection of film defects

In this section we provide the effectiveness and efficiency to detect film defects. To show

the effectiveness, the precision and recall were evaluated for defect detection with respect

to various thresholds. We first need to determine two parameters, � for obtaining the

boundary value between a defect and background, and k for computing the threshold for a

defect region. Parameter � tends to depend on the size of the defect within a film image,

and is determined adaptively. The appropriate range (0.2�� 0.5) in which the proposed

method shows superior performance can be derived based on experiment using the training

sets. Our method performs well when the value of k is approximately 4.2. Therefore, the

optimised parameters were used for subsequent experiments.To demonstrate the superiority of our method, it was compared with the well-known

adaptive threshold method proposed by Kim et al. (2004). Since Kim et al.’s method uses

parameter k only, the experimental result is presented with respect to k. For fairness, an

optimised value of parameter k for Kim et al.’s method was also obtained using

the training sets for which the method shows the best performance. As a result, it

performs best at approximately k¼ 5.0. Therefore, the range for comparison was

determined to be 2.0� k� 6.0, which contains the best-performing intervals of both

methods. Figure 12 shows the experimental results in terms of the precision and recall

using the validation sets.As shown in Figure 12, the precision is sensitive to the change in k. It increases rapidly

with increasing k while the recall decreases slowly. It is important to consider both

measures simultaneously. The proposed method shows the best performance (preci-

sion¼ 1.0, recall¼ 0.99) in the range 3.6� k� 4.5, whereas Kim et al.’s method shows the

best performance ( precision¼ 0.94–0.96, recall¼ 0.95–0.97) in the range 4.8� k� 5.4. As

shown in the figure, the proposed method outperforms Kim et al.’s method for most of the

range of k.

Table 2. Five sample defects for each type.

Type Sample defects #

Defects

Dent 28

Scratch 38

Black spot

30

White spot

48

Swollen defect

43

Crater 25

Unknown 6

1 2 3 4 5



The above experiment has another important implication. In reality, there are variousrecall-sensitive applications in the defect inspection industry, in which the recall should be1.0, i.e. no false dismissal is allowed. In Figure 12, when the recall is maintained at 1.0, theprecision of the proposed method is up to 0.94, whereas that of Kim et al.’s method was upto 0.57. This shows our method to be effective in recall-sensitive applications as well as inapplications that consider the precision and recall equally.

The execution time for detecting defects was measured to evaluate the efficiency.Figure 13 shows the efficiency of the two methods depending on the size of the defectimages, where the size is represented by the number of pixels. Finding defects should beperformed in a real-time environment since the production line runs continuously withoutstopping. The real-time constraints differ according to the characteristics of the productionline, such as the moving speed of the film, image resolution, and film size covered by

Figure 12. Comparison of the effectiveness for detecting defects with respect to k usingvalidation sets.

Figure 13. Efficiency of detecting film defects depending on the size of the defect images.



the camera. The real-time constraint of our production environment was that film imagesof 300� 300 and 500�500 pixels should be processed within approximately 0.25 and 0.9seconds, respectively. Our method exhibits 0.076 and 0.203 seconds, respectively,outperforming the real-time constraint. As shown in the comparison with Kim et al.’smethod, the execution time for detecting defects is delayed slightly over most of the rangeof defect image sizes, which can be negligible considering the real-time constraints. Thedelay in execution time is attributed to the computation of � and is a trade-off for bettereffectiveness. Another promising observation is that the execution time increases linearlywith increasing image size. It is expected that our method can achieve a faster processingtime if the inspection machine is equipped with a better hardware configuration, such as afaster CPU, memory, and grabber boards.

4.3 Classification of film defects

This section presents the effectiveness of our method in determining the types of filmdefects by evaluating the precision and recall. For large-sized defects, such as craters,l-dents and l-swollen defects, our method was compared with the existing shape-contextmethod.

First, it is important to determine some of the experimental parameters mentioned inSection 3.2 using the training sets. If thin_ratio or adr_ratio is less than 0.4 it can beclassified as a scratch defect. To classify small-sized defects, such as black and white spots,s-dents, and s-swollen defects, wt¼ 0.1 and bt¼ 0.9 were used. These values are determinedempirically. In order to identify large-sized defects including craters, l-dents and l-swollendefects, the weight !¼ 0.5 was used for the two terms CS and CI, which means equalimportance. In addition, it is important to obtain a threshold TH in order to determineany unknown type during cost matching. An experiment was carried out to obtain anappropriate threshold using the training sets, and TH¼ 0.55 was obtained, which will beused in the subsequent experiment. If the matching cost between a given image and anyreferential defect image in a template database is larger than 0.55, it is then regarded as anunknown type.

Figure 14 shows the change of precision in classifying large-sized defects using theshapeþID and shape-context descriptors with respect to TH, using the validation sets. Forthe shapeþID descriptor, the precision increased with increasing TH, but began to decreasewhen the threshold reached approximately 0.55. This means that our method shows thebest precision of 0.83 at that position. For the shape-context method, the precisiondecreased with increasing TH, showing the best precision of 0.62 at TH¼ 0.40. As shownin the figure, our method outperforms the shape-context method in most threshold ranges.The use of the intensity distribution in addition to the shape descriptor enhances theaccuracy in classifying large-sized defects considerably.

The precision and recall for determining all types of defects are presented since allexperimental parameters have been determined to classify film defects. Table 3 shows theexperimental results indicating the precision and recall for determining the type of defect.In the first column of the table, the term ‘ground truth’ indicates the classification resultsperformed by domain experts. This is regarded as the correct classification. The term‘correct’ means that the correct result has been derived by our experiment while the term‘incorrect’ means an incorrect result. The overall results show precision¼ 0.78 andrecall¼ 0.78, respectively, which are comparatively high and useful in a practical



production environment. In the case of dents and swollen defects, which occur frequently,the result demonstrates high effectiveness, showing precision¼ 0.94–0.96 and recall¼ 0.84–0.86, respectively. Classifying the scratch type shows a relatively low effectiveness, whichneeds more investigation in the future. Unknown types also need to be investigated sincethey degrade the overall performance. The current classification of defect types is notstandardised in related industries and there is also a different naming convention amongcompanies, institutes, and laboratories. Considerable effort should be made to classify theunknown types of defects in detail and maintain a consistent naming of those defect types.This will clearly enhance the effectiveness of the classification.

4.4 Scale and rotational invariance

The last experiment is to determine if our method based on the shapeþID descriptor isrobust with respect to scale and rotational transformation. A set of test defect images wasprepared to evaluate the scale and rotational invariance. Twelve defect images were first

Figure 14. Precision of classifying large-sized defects using shapeþID and the shape-contextdescriptors with respect to TH.

Table 3. Precision and recall for determining the type of defect.

Type of defect image

TotalScratch CraterWhitespot

Blackspot

Swollendefect Dent Unknown

Ground truth 25 17 32 20 29 19 4 146Experimental resultsCorrect 14 16 29 13 25 16 1 114Incorrect 9 5 7 0 1 1 9 32

Precision 0.61 0.76 0.81 1.00 0.96 0.94 0.10 0.78Recall 0.56 0.94 0.91 0.70 0.86 0.84 0.25 0.78



selected from the large-sized images. Each was transformed by rotating 45�, 90�, 135�, and180�, and by scaling up and down randomly. Hence, a set of 60 test images was obtained.Figure 15 shows an original crater image and four transformed images as an example.

The evaluation procedure is as follows. Given a query defect image, the systemretrieves the top five similar defect images out of 60 images using the k-NN (k-nearestneighbor) search method. The cost matching function based on the shapeþID descriptor isused to measure the similarity between two defect images. In this way, queries are issued 60times, once by a test image. Each query produces a set of five answer images (5-NN),resulting in a set of 300 answer images. Each answer image is examined to determine if it isa rotated and/or scaled form of the query image. If this is the case, it is regarded as acorrect result. As an experimental result, we have 286 correct results out of 300 retrieveddefect images (precision¼ 0.96). From the high precision, it can be concluded that ourmethod using the shapeþID descriptor is robust with respect to scale and rotationaltransformation.

5. Conclusions

The defect inspection of polarised films is becoming increasingly important in themanufacture of flat display panels. In this paper we have proposed a defect inspectionsystem for TFT-LCD film images that locates defects in a real-time environment anddetermines the type of defects. Our system supports the classification of six types ofdefects, such as dents, scratches, black and white spots, swollen defects, and craters, whichoccur frequently in a production line, and thus have a significant impact on the quality ofthe final product. The proposed system was designed to locate defects promptly using anadaptive threshold technique and determine the defect types through image analysis usingvarious features, such as the geometric characteristics and the shapeþID descriptor.

To evaluate the effectiveness and efficiency of our proposed system, variousexperiments were carried out using a set of test images obtained in a real productionline. First, the method identifies defects efficiently and effectively enough to be used in areal-time environment (precision¼ 1.00, recall¼ 0.99). Next, it classifies the type of defectwith considerable correctness (precision¼ 0.78, recall¼ 0.78) for various types of defects.Finally, it demonstrates scale and rotational invariance (precision¼ 0.96), which is adesirable property for a defect inspection system.

Our method was used to inspect film defects, but it is believed that other potentialapplication domains can also benefit. As future work, we plan to investigate the defectclassification for more specialised defects using micro- and macro-film images. Most of thecurrent work focuses on classifying a few types of defects, such as spots, scratches,

(e) 180°-enlarged (a) An original image (b) 45°-enlarged (c) 90°-shrunk (d) 135°-shrunk

Figure 15. An original crater image and four transformed images.



and muras. There are diverse unknown types of defects in reality and new types arecontinuously being found as new cameras and illumination equipment are introduced.However, there are few studies on these defects. Future work will analyse and classifythose defects in detail. In addition, more study will be carried out to improve theeffectiveness of classifying defects by applying various pattern recognition techniques.Finally, we plan to apply the proposed method to other application domains consideringtheir unique characteristics, such as flaw detection for vessels, instruments and fruit, andface or gesture recognition.

Acknowledgements

This work was supported by Defense Acquisition Program Administration and Agency for DefenseDevelopment under the contract UD030000AD.

References

Baek, S.I., et al., 2004. Inspection of defect on LCD panel using polynomial approximation. In:Proceedings of the IEEE region 10 conference, 235–238.

Belongie, S. and Malik, J., 2000. Matching with shape contexts. In: Proceedings of the IEEE

workshop on content-based access of image and video libraries (CBAIVL’00).Jiang, B.C., Wang, C.C., and Liu, H.C., 2005. Liquid crystal display surface uniformity defect

inspection using analysis of variance and exponentially weighted moving average techniques.International Journal of Production Research, 43 (1), 67–80.

Joncker, R. and Volgenant, A., 1987. A shortest augmenting path algorithm for dense and sparselinear assignment problem. Computing, 38 (4), 325–340.

Kim, W.S., et al., 2004. Detection of spot-type defects on liquid crystal display modules.

Key Engineering Materials, 270–273 (1), 808–813.Lee, J.Y. and Yoo, S.I., 2004. Automatic detection of region-mura defect in TFT-LCD. IEICE

Trans. Inf. & Syse, E87-D (10), 2371–2378.

Lu, C.J. and Tsai, D.M., 2004. Defect inspection of patterned thin film transistor-liquid crystaldisplay panels using a fast sub-image-based singular value decomposition. InternationalJournal of Production Research, 42 (20), 4331–4351.

Michael, D.H., et al., 1997. A robust visual method for assessing the relative performance of edge-

detection algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19 (12),1338–1359.

Mori, G., Belongie, S., and Malik, J., 2005. Efficient shape matching using shape contexts. IEEE

Transactions on Pattern Analysis and Machine Intelligence, 27 (11), 1832–1837.Oh, J.H., et al., 2004. Line defect detection in TFT-LCD using directional filter bank and adaptive

multilevel thresholding. Key Engineering Materials, 270–273 (1), 233–238.

Tsai, D.M., Chuang, S.T., and Tseng, Y.H., 2007. One-dimensional-based defect inspection ofmultiple patterned TFT-LCD panels using Fourier image reconstruction. International Journalof Production Research, 45 (6), 1297–1321.

Tsai, D.M. and Hung, C.Y., 2005. Automatic defect inspection of patterned thin film transistor-liquid crystal display panels using one-dimensional Fourier reconstruction and waveletdecomposition. International Journal of Production Research, 43 (21), 4589–4607.

Yoon, Y.G., et al., 2008. An effective defect inspection system for polarized film images using image

segmentation and template matching techniques. Computers & IE, 55 (3), 567–583.Zhang, Y. and Zhang, J., 2005. A fuzzy neural network approach for quantitative evaluation of

mura in TFT-LCD. In: Proceedings of the international conference on neural networks and

brain, 424–427.



Documents

International Journal of Production Research An effective and efficient …islab.kaist.ac.kr/chungcw/tft-lcd.pdf · 2015. 11. 26. · International Journal of Production Research