License Plate Detection and Recognition Algorithm

for Vehicle Black Box

Jung-Hwan Kim, Sun-Kyu Kim, Sang-Hyuk Lee, Tae-Min Lee and Joonhong Lim

Department of Electronic Systems Engineering, Hanyang University

Seoul, Republic of Korea

{ angelkim88, jhlim } @hanyang.ac.kr

Abstract— Almost every vehicle has currently installed black

box since the stored images by black box can be used to investigate

the exact cause of the accident. One of the most important aspects

in an accident investigation is the license plate detection and

recognition as the license plate has information about the driver

and car. This paper presents a novel algorithm for license plate

detection and recognition using black box image. The proposed

license plate recognition system is divided into three stages: license

plate detection, individual number and character extraction, and

number and character recognition. The Gaussian blur filter is

used to remove noise in the image and then we detect the license

plate edge using modified Canny algorithm. Second, we determine

license plate candidate image using morphology and support

vector machine. Finally, we recognize the numbers and characters

using k-nearest neighbor classifier. The experimental study results

indicate that the license plate detection and recognition algorithm

has been successfully implemented.

Keywords—License plate; Vehicle Black Box; Support vector

machine; k – nearest neighbor; Image Processing;

I. INTRODUCTION

A license plate detection and recognition is one of important processes in investigating a car accident since license plate has information about driver and vehicle identification. Currently, almost all vehicle black boxes store only driving images so it is not easy to detect and recognize license plate correctly. To overcome this problem, we propose an algorithm that automatically recognizes license plate using a vehicle black box image in this paper. Fig. 1 shows the Korean license plate and Table 1 shows the format. The new license plate format is made up of ## (letter) #### where # is a number. This license plate is available in a size format similar to the license plates used in the European Union, 520 mm wide by 110 mm tall. The color scheme is a simple design which is black and white[1]. This paper uses these features to detect license plate.

TABLE I. FORMAT OF KOREAN LICENSE PLATE

Position Classification Symbol

Two initial

digits

Passenger vehicles 01 ~ 69

Vans 70 ~ 79

Freight Vehicles 80 ~ 97

Specialized Vehicles 98 ~ 99

Letter Private Vehicles 가~마, 거~저, 고~조, 구~주

Rental Vehicles 허, 하, 호

Four final digits

- 0100 ~ 9999

Fig. 1. Feature of Korean license plate.

Fig. 2 shows a flow chart of the proposed algorithm. License plate recognition system is divided into three stages: license plate detection, individual character and number extraction, and number and character recognition[2-5]. First of all, we use Gaussian blur filter to remove noise from the image and then we detect the license plate edge using modified Canny algorithm. We determine the license plate candidate image using closing operator and SVM. The SVM performs a supervised learning about 1000 times. Second, we use Korea regulations of the license plate to find individual letters and numbers. Lastly, we recognize the letters and numbers using k-NN classifier. The experimental study shows that the license plate detection and recognition algorithm has worked successfully. This algorithm may be useful for accurate accident investigation and safe driving.

Fig. 2. Flow chart of license plate detction and recognition algorithm.

This research was supported by Ansan-Si hidden champion fostering and

supporting funded by Ansan city.

II. LICENSE PLATE DETECTION

A. Distortion correction

In general, distortion correction is necessary to detect the vehicle license plate because a camera image is distorted by the lens. In particular, the vehicle black box images have experienced severe distortions because the black box uses a wide angle lens to reduce blind spots. Fig. 3 shows image distortions by barrel and pincushion.

Fig. 3. Examples different type of lens distortion.

Distortion types by concave and convex lens are spectrum, decentering and radial distortion. The radial distortion is considered as a more significant part of lens distortion correction. Radial distortion can be corrected by minimizing the intersection ratio between the coordinates and straight lines of four points on the same line. This method can resolve the distortions by warping the image with a reverse distortion and determining which distorted pixel corresponds to each undistorted pixel[6-7].

The black box images have noises from various reasons. Image noise usually occurs in the process of converting light into voltage in an image input device using CCD. This can be generated by the circuits of the digital camera, temperature or other causes. This noise should be removed because it can cause errors in detecting the license plate. In this paper, we use a Gaussian blur filter as a pre-processing step to remove noise.

Fig. 4. Template for the 5×5 Gaussian blur filter.

The Gaussian blur filter uses a Gaussian function for image blurring by applying the transformation to each pixel in the image[6-9]. The Gaussian function equation in two dimensions can be expressed as

𝐺(𝑥, 𝑦, 𝜎) =1

2𝜋𝜎2𝑒

−𝑥2+𝑦2

2𝜎2 (1)

where 𝑥 and y is the distance from the origin the distance in the horizontal and vertical axis respectively. σ is the standard deviation of the Gaussian distribution. The Eq.(1) produces a surface whose contours are concentric circles with a Gaussian distribution from the center point when applied in two

dimensional plane. The values from this distribution are used to form a convolution matrix which is applied to the original image. This distribution is shown in Fig. 4. A new value of each pixel is set to a weighted average of that pixel neighborhoods. The neighboring pixels receive smaller weights as their distance to the original pixel increases. On the other hand, the original pixel value receives the heaviest weight. As a result, the Gaussian blur filter can reduce random noise of black box image .

B. License plate edge detection

There are various methods for detecting edge of license plates[10-13]. In this paper, we detect the license plate using the rectangle shape feature. The license plate can be founded by detecting a horizontal and vertical edge because of rectangle shape. There are several methods for detecting vertical and horizontal edges such as Sobel and Prewitt. To reduce the influence of noise and detect the edge of the vertical component, we propose a modified Canny method in this paper. This detector is formulated with three criteria which are minimum error rate, location accuracy and edge thickness[14]. The Canny algorithm uses a variant calculation to satisfy these requirements. The process of modified Canny edge detection algorithm based on three criteria is divided into 3 different steps as follows[15];

(1) Find the direction and the intensity of image gradient using vertical type mask.

(2) Apply non-maximum suppression to get rid of spurious response to edge detection.

(3) Track edge by hysteresis : Finalize the detection of edges by suppressing all the other edges that are weak and not connected to strong edges.

The first step is different from the edge detection algorithm in which the vertical and horizontal edges are detected. In this paper, the Canny algorithm is modified to find only the vertical edges since the vertical edge is used to find the license plate. The following mask is used for the vertical edge detection method :

𝐺𝑦 = [−1 0 1−2 0 2−1 0 1

] (2)

The modified edge detection operator returns a value for the first derivative of the vertical direction 𝐺𝑦 as in

𝐷(𝑥, 𝑦) = arctan (𝑑𝑥/𝑑𝑦) (3)

∇𝑓 = (𝜕𝑓

𝜕𝑥,

𝜕𝑓

𝜕𝑦) = (𝑑𝑥 , 𝑑𝑦) (4)

𝑆(𝑥, 𝑦) = √𝑑𝑥2 + 𝑑𝑦

2 (5)

where ∇𝑓 is edge gradient, 𝐷(𝑥, 𝑦) is direction, and 𝑆(𝑥, 𝑦) is magnitude.

The second step is to make the thin edge. The edge extracted from the gradient value may be quite blurred after applying gradient calculation. With respect to the third criterion, there has to be only one accurate response to the edge. We apply non-max suppression to suppress all gradient values to zero except the local maximum. The image is scanned along the direction of the image gradient and set to zero if the pixel is not part of the local

maximum. This method has the effect of suppressing all image information that is not part of local maxima.

After application of non-maximum suppression, remaining edge pixels provide a more accurate representation of real edges in an image. However, some edge pixels remain that are caused by noise variation. The third stage filters edge pixels with a weak gradient value and preserve edge pixels with a high gradient value in order to describe these spurious responses. The tracking process exhibits hysteresis controlled by two thresholds that are 𝑇ℎ𝑖𝑔ℎ and 𝑇𝑙𝑜𝑤. Tracking can only begin at a point on a ridge

higher than 𝑇ℎ𝑖𝑔ℎ. Tracking then continues in both directions out

from that point until the height of the ridge falls below 𝑇𝑙𝑜𝑤 . This hysteresis helps to ensure that noisy edges are not broken up into multiple edge fragments[8, 15]. Finally, the vertical edge of the license plate can be found.

C. Morphology

The license plate area is long horizontal shape in the black box image. In order to detect the license plate, the long horizontal shape should be enclosed in one area. This can be implemented by performing a closing operation of morphology. A morphology operation is performed as a pre-processing step to detect license plate[8-11]. Morphological operators take a binary image and a structuring element as inputs and combine them using set operators such as intersection, union, inclusion and complement. These set operators process objects in the input image based on the characteristics of the encoded shape in the structuring element. The structuring element is shifted over the image and then at each pixel of the image its elements are compared with the set of the underlying pixels. If the two sets of elements match the condition defined by the set operator, the pixel from the origin of the structuring element is set to be a pre-defined value. A morphological operator is therefore defined by its structuring element and the applied set operator. The basic Morphological operators are dilation and erosion[6-7].

Typically, the dilation operation uses a structuring element to expand and probe the shapes contained in the input image. The morphological transformation dilation combines two sets using vector addition. The dilation is the point set of all possible vector additions of pairs of elements, one from each of the sets A and B. Let E be a Euclidean space or an integer grid, then A a set of binary image in E, and B a set of structuring elements are

regarded as a subset of an Euclidean space 𝑅𝑑 for some dimension d . On the other hand, the erosion operator combines two sets using vector subtraction of set elements. Thereby erosion operation reduces the size of the object in the image and enlarges the background.

The closing operation consists of dilation followed by erosion. The closing operation connects objects that fill up small holes close to each other and smooths the object outline by filling up narrow deviations. The closing of an image A by the structuring element B is denoted by A⋅ B and is defined as

A ⋅ B = (A ⊕ B) ⊖ B . (6)

The effect of the operator is to preserve background regions that have a similar shape to this structuring element, or that can completely contain the structuring element, while eliminating all

other regions of background pixels. In this paper, The area of license plate is detected using the closing operation property.

After the license plate area is found by the closing operation, the outer line can be detected and then used to find the rectangle of the minimum area. The rectangles of detected license plate candidate images are subject to geometric distortion introduced by perspective irregularities since the position of the camera with respect to the scene changes the apparent dimensions of the scene geometry[12].

Fig. 5. Affine transformation of license plate.

The affine transformation applied to a uniformly distorted image can correct a range of perspective distortions by transforming the measurements from the ideal coordinates to those actually used as shown in Fig. 5. The affine transformation is written in homogeneous coordinates as

[𝑐𝑜𝑠𝜃 −𝑠𝑖𝑛′𝜃 0𝑠𝑖𝑛𝜃 𝑐𝑜𝑠𝜃 0

0 0 1

] ∙ [𝛼 0 00 𝛽 00 0 1

] ∙ [1 0 𝑡𝑥

0 1 𝑡𝑦

0 0 1

] (7)

where 𝑐𝑜𝑠𝜃 and 𝑠𝑖𝑛𝜃 are in rotation matrix, 𝛼 and 𝛽 are in resize matrix, and 𝑡𝑥 and 𝑡𝑦 are in translation matrix. The

rotation angle and the center coordinates can be substituted into the formula to correct the plate candidate area to be arranged in a horizontal rectangle. After correcting the license plate candidate area, candidates are detected again using the width and the section ratio of the detection area[13]. The width and aspect ratio is determined by taking into consideration the characteristics of the license plate. The rotation angle and the center coordinates can be substituted into the formula to correct the plate candidate area to be arranged in a horizontal rectangle.

D. Support Vector Machine

If the license plate area is found only by using the size and the section ratio, it cannot be detected accurately because objects with similar size and section ratio are detected together. There are various classification algorithms to solve this problem[16-18]. In this paper, SVM, which is one of the machine learning algorithms, is used to detect the license plate area.

Optimal classification of a separable two class problem is achieved by maximizing the width of the margin between the two classes. This width is defined as the distance between the discrimination hyper-surfaces in n-dimensional feature space. Vectors from each class that are closest to the discriminating surface are called support vectors. The support vectors thus specify the discrimination function. Fig. 6 (a) shows which is non-optimal linear discrimination and Fig. 6 (b) shows that an optimal linear discriminator which maximizes the margin between of the two classes by SVM.

(a) (b)

Fig. 6. Two classifications of SVM. (a) Non-optimal linerar discrimination,

(b) Optimal linear discrimination.

The maximizing margin is equivalent to the optimization problem given by

minimize J(𝜔) =1

2∥ 𝜔 ∥2,

Subject to 𝑦𝑖(𝜔𝑇𝑥𝑖 + b) ≥ 1 ∀ i (8)

Here, 𝐽(𝜔) is a quadratic function which means that there exists a single global minimum and no local minima. To solve this problem, we define Lagrangian as

𝐿𝑃(𝜔, 𝑏, 𝛼) =1

2∥ 𝜔 ∥2− ∑ 𝛼𝑖[𝑦𝑖(𝜔𝑇𝑥𝑖 + 𝑏) − 1]

𝑁

𝑖=1

(9)

The constrained minimization of J(𝜔) = 1/2 ∥ 𝜔 ∥2 is solved by Lagrangian technique. Finally, the license plate can be detected among candidate areas by support vector.

III. LICENSE PLATE NUMBER & LETTER RECOGNITION

A. License Plate Image normalization

To improve the license plate recognition, Preprocessing is required to classify area of numbers and letters. In this paper, we perform five steps for preprocessing as follows;

[Step1] The license plate image size is changed to 180 ×35 and image binarization is performed.

[Step2] Remove 3 pixels up and down and 6 pixels left and right.

[Step3] Obtain the minimum rectangular area of the number and letter using the outline line.

[Step4] The third region from the left is classified as a character and OR operation is performed.

[Step5] The classified objects are moved to center of the image.

The image resizing and binarization in the first step is to normalize the license plate image. This normalization increases the number and letter recognition rate. The reason for the second step is that there are a lot of noises in the upper, lower, left, and right corners of the license plate. The third step is to remove the small noises that have not been removed in the second stage. The fourth step is combining the divided letter areas into one object using the OR operation. The letter area can be detected by separating into two or more areas due to the shape of Korean

language. In addition, the third part of Korean license plate is a letter as in Fig. 7 (a). The classified objects must be in the same position to increase the accuracy and effectiveness of learning so the object is centered using the histogram in the fifth step as shown in Fig. 7 (b).

(a) (b)

Fig. 7. Pre-Processing to recognize numbers and letters of the license Plate. (a) minimum areas, (b) Horizontal and vertical histogram measurement.

B. k – Nearest Neighbor Classification

There are many ways to recognize classified objects. In this paper, we use a k-NN classifier to recognize letter and numbers. The k-NN is a type of instance based learning where the function is only approximated locally and all computations are deferred until classification. This classifier algorithm is one of the simplest of all machine learning algorithms. In k-NN classification, an object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors. If k = 1 then the object is simply assigned to the class of that single nearest neighbor as shown in Fig. 8.

Fig. 8. Example of k-NN classification.

IV. EXPERIMENT RESUILTS

In this paper, we use the black box images taken between 12 and 15 o'clock to detect the license plates. The distance from the license plate is less than 3m. The experiment has been performed on Microsoft visual studio community 2015 using OpenCV3.1. Processor specifications of computer are Intel(R) Core(TM) i7-4790 and 8GB RAM. CPU clock is 3.6 GHz and operating system is Windows 8.1K 64Bit. The data sets used to recognize the numbers and letters on the license plate are shown in Figs. 9 (a) and (b). The license plate and non-license plate data sets for SVM learning are shown in Figs. 9 (c) and (d).

(a)

(b)

(c)

(d)

Fig. 9. Data sets for license plate detection and recognition. (a) 400 numbers

(10 classes, 0~9), (b) 5120 letters (40 classes) (c) 100 license plates (d) 112

non-license plates

Processing time of the proposed algorithm is approximately 53.66 msec. This time can be measured using ‘getTickCount’ function of windows API. The experimental results are shown in Fig. 10. Fig. 10 (a) is an image in which the distortion of the lens is corrected. We use chessboard image to measure necessary correction parameters. After correcting the distortion

of the black box image, the image is converted to gray scale. In addition, 5 × 5 Gaussian blur filter is applied to remove noises. The resolution of image in Fig. 10 (b) is 1280 × 720. As shown in Fig. 10 (c), the modified canny edge algorithm is used to detect only vertical edges. To find the license plate candidate areas from the edge image, we use the closing operation as shown in Fig. 10 (d). The license plate candidate areas are preferentially discriminated by the area and the size as in Fig. 10 (e). In order to compare these candidate areas, the normalization process is performed as in Fig. 10 (f). The size of the candidate image is adjusted to 180 × 35. Finally, the license plate area can be detected by using SVM. The detected area is shown in Fig. 10 (g). The number and letter are recognized by using k-NN in the detected area as in Fig. 10 (h) where the number of k is 15.

(a) (b)

(c) (d)

(e) (f)

(g) (h)

Fig. 10. The license plate detection and recognition result. (a) Undistortion (b) Gray scale image (c) Modified Canny edge (d) closing operator (e) license plate

candibate areas (f) Affine transformation (g) License plate detection using

SVM (h) letter and numbers recognition using k-NN

Table 2 summarizes confusion matrix of the license plate detection. There are 26 photos without license plates and 121 photos with license plates as in Table 2. A total of 147 photographs are used in the experimental study. Precision rate is 97.32%, recall rate is 90.08%, and accuracy is 89.79%. Here, the precision is the fraction of relevant instances among the retrieved instances, while recall is the fraction of relevant instances that have been retrieved over total relevant instances in the image.

TABLE II. CONFUSION MATRIX OF THE LICENSE PLATE DETECTION

Total 147 images Predicted

Plate Not Plate

Actual Plate 109 (TP) 12 (FN)

Not Plate 3 (FP) 23 (TN)

V. CONCLUSION

The automatic license plate recognition is one of the most important aspects in an car accident investigation since the license plate has information about the driver, car and official identification numbers. In this paper, we propose a detection and recognition algorithm of a license plate using black box image and perform experimental study to implement the algorithm. The proposed algorithm has detected the edge of the plate well. This algorithm detects only vertical edge of license plate by modifying Canny method. In addition, the success rate of the proposed algorithm is close to 90% which is satisfactory considering that a small amount of data sets is used. The success rate can be increased if we use more data sets. This algorithm may be useful for safe driving and accurate accident investigation. The future study may include increasing the recognition rate with the enhanced algorithm and collecting various data sets.

REFERENCES

[1] https://en.wikipedia.org/wiki/Vehicle_registration_plates_of_South_KorKo

[2] Amitava Choudhury, Alok Negi, “A New Zone Based Algorithm for Detection of License Plate from Indian Vehicle”, PDGC conf. , pp. 370-374, Dec 2016.

[3] Priyanka Prahakar, P. Anupama, “A Novel Design For Vehicle License Plate Detection and Recognition”, ICCTET conf. , pp. 7- 12, Dec 2014.

[4] Muayad Ali Hamood Bakhtan, Munaisyah Abdullah, Aedah Abd Rahman, “A Review on License Plate Recognition System Algorithms”, ICICTM conf. , pp. 84-89, May 2016.

[5] Ihsan Ullah, Hyo Jong Lee, “An Approach of Locating Korean Vehicle License Plate Based on Mathematical Morphology and Geometrical Features”, CSCI conf. , pp. 836-840, Dec 2016.

[6] Milan Sonka, Vaclav Hlavac, Roger Boyle, Image Processing Analysis, and Machine Vision. Fourth edition, Boston, CA: Cengage Learning, 2015.

[7] Mark Nixon, Feature Extraction & Image Processing for Computer Vision. Third edition, Cambridge, CA: Academic Press, 2012

[8] Habib Zaidi, Quantitative Analysis in Nuclear medicine imaging. First edition, Berlin, CA: Springer, 2006

[9] Chao Gou, Kunfeng Wang, Bo Li, Fei-yue Wang, “Vehicle License Plate Recognition Based on Class-specific ERs and SaE-ELM”, ITSC conf. , pp 2956-2961, Oct 2014.

[10] Chao Gou, Kunfeng Wang, Yanjie Yao, Zhengxi Li, “Vehicle License Plate Recognition Based on Extremal Regions and Restricted Boltzmann Machines”, IEEE Transactions on Intelligent Transportation System, pp. 1096-1107, November 2015.

[11] Naiguo Wang, Xiangwei Zhu, Jian Zhang, “License Plate Segmentation and Recognotion of Chinese Vehicle Based on BPNN”, CIS conf. , pp. 403-406, Dec 2016.

[12] Animesh Chandra Roy, Muhammad Kamal Hossen, Debashis Nag, “License Plate Detection and Character Recognition System for Commercial Vehicles based on Morphological Approach and Template Matching”, ICEEICT conf. , pp. 1-6, Sep 2016.

[13] Hitesh Rajput, Tanmoy Som, Soumitra kar, “An Automated Vehicle License Plate Recognition System”, IEEE Journals & Magazines Volume 48, pp. 56-61, Aug 2015.

[14] Canny, J., “A Computational Approach To Edge Detection”, IEEE Trans. Pattern Analysis and Machine Intelligence, pp. 679-698, Nov 1986.

[15] Rachid Deriche, “Using Canny's criteria to derive a recursively implemented optimal edge detector”, International Journal of Computer Vision, pp. 167-187, June 1987.

[16] Tao Hong, Anilkumar Kothalil Gopalakrishnam, “ License Plate Extraction and Recognition of a Thai Vehicle Based on MSER and BPNN”, KST conf. , pp 48-53, Jan 2015.

[17] Asmaa Elbamby, Elsayad E. Hemayed, Dina Helal, Mohamed Rehan, “Real-time Automatic Multi-Style License Plate Detection in Videos”, ICENCO conf. , pp. 148-153, Dec 2016.

[18] Worawut Yimyam, Mahasak Ketcham, “The Automated Parking Fee Calculation Using License Plate Recognition System”, ICDAMT conf. , pp. 325-329, March 2017.