An Embedded System for Real-Time Traffic Sign Recognizing

M. A. SOUKI1,2, L. BOUSSAID1,2, M. ABID1,3 Email: Souki.Amin@gmail.com , Lotfi.Boussaid@enim.rnu.tn , Mohamed.Abid@enis.rnu.tn

1. Unité de recherche CES, 2. Ecole Nationale d’Ingénieurs de Monastir, 3. Ecole Nationale d’Ingénieurs de Sfax

Abstract –-Automated recognition of traffic signs is an important issue for driver assistance and autonomous navigation systems. Those systems have to be fast and robust to detect signs in real-time and recognize them precisely. In this paper, we present a HW/SW system for detection and recognition of circular traffic signs mainly based on color and shape properties. The designed system has been implemented in a CYCLONE II DE2 Board using the Nios II softcore processor which is characterized by its flexibility and programmability.

Keywords: Traffic sign detection, Recognition, FPGA, Nios-II, Hough Transform

I. INTRODUCTION Over the world, 1.2 million people were killed in traffic

crashes in 2002, which was 2.1% of all global deaths and the 11th ranked cause of death [1].

Nowadays, we are moving towards a new era in which, thanks to technologies, crashes are rare rather then commonplace. In fact, new Intelligent Transport Systems (ITS) have been introduced in automotive industry in order to save money and lives, and to make the driving safe and convenient. The new challenge of those systems is to integrate new smart sensors which can provide information about the current traffic situation on the road, show the danger and difficulties around the drivers, give warnings to them, and help them when driving along the road.

One of the important fields in the ITS is the Road and traffic sign recognition. Actually, many research groups and companies were interested in this field, and enormous amount of work has been done.

In this way, different techniques have been used, and big improvements have been achieved during the last decade. The identification of the road signs is achieved by two main stages: detection, and recognition. In the detection phase, the image is pre-processed, enhanced, and segmented according to the sign properties such as color or shape. The output is a segmented image containing potential regions which could be recognized as possible road signs.

In the recognition stage, each of the candidates is tested against a certain set of features (a pattern) to decide whether it is in the group of road signs or not, and then according to these features they are classified into different groups.

The contribution of this paper consists on using new technology based-platform equipped by a Cyclone II FPGA and the embedded NIOS-II soft core processor in order to implement a flexible and robust real-time traffic sign recognition system (TSR). The proposed approach is based on color information and shape techniques for detection and recognition of circular traffic signs.

II. RELATED WORKS A traffic sign recognition system usually involves two

main steps: a detection of potential traffic signs in the image based on the common shape and/or color design of sought traffic signs, and classification of the selected regions of interest (ROIs) for identifying the exact type of sign, or rejecting the ROI.

The most essential types of road signs are restricting, warning and information (or guidance) signs. By their properties those signs can be separated into two groups. First group consists of “red-bordered” signs (warning and restricting) for which a big amount of methods were developed. However the second group which includes information signs is not so well-studied because they are not as well-structured as warning and restricting signs [2].

The majority of TSR methods are based on color information (e.g. [3], [4] and [5]), which makes the detection step easier. This information should be utilized effectively and efficiently even in the knowledge that color and shape vary with the change of lighting conditions and viewing angles. However, most color-based techniques run into problems if the illumination source varies not only in intensity but also in color as well. This is because that the spectral composition, and therefore the color, of daylight change depending on weather conditions, e.g., sky with/without clouds, time of the day, and night when all sorts of artificial lights are surrounded [6].

In the other hand, shape-based methods [7], [8] and [9] in grayscale space permit correct operation in dark or night condition, or robustness for detection of signs with colors faded away by time.

For the classification task, most approaches utilize well studied techniques of classification schemes such as template matching [8], multi-layer perceptrons [10], radial basis function networks [11], Laplace kernel classifiers [12], etc.

III. DETECTING SIGN CANDIDATE REGIONS In this stage, we use a simple and robust method to detect

circular panels. Actually, we aim to implement approaches that can be reliable and ensure real-time computation in order to obtain decision at the appropriate time. Thus, our optimizations will concern image preprocessing, edge detection and Hough Transform technique steps (figure 1).

For smoothing filter, we will use an average filter. The choice of this filter is due to its simplicity on comparison with the median filter.

Figure 1. Basic steps in a traffic sign detection algorithm

A. Image Pre-processing Before the sign is detected, it is passed through a number

of preprocessing steps. The preprocessing is done mainly to remove redundant and irrelevant information to simplify the task for further processing such as edge detection.

The first step in preprocessing procedure consists on applying an average filter in order to reduce noise and to make image more homogeneous and easily discernible.

The second step concerns the red segmentation which is introduced in [13]. In fact, the red color represents an interesting alternative and is definitely an important attribute for road signs to facilitate and improve driving conditions. Therefore, the use of this propriety for segmentation gives an advantage to minimize the research on candidates regions.

To realize the image segmentation, each input pixel must be tested according the following three conditions:

(Ri> Gi) and (Ri−Gi ≥ ΔRG) and (Ri−Bi ≥ ΔRB) (1)

If the three conditions are verified, the output takes a one, otherwise it takes a zero. The choice of the threshold ΔRB and ΔRG is carried out in an empirical way.

After that, the obtained binary image is passed through a simple gradient filter for edge detection in two direction x and y. The advantage of this filter is that it requires less addition and multiplication operators than complex filters such as Canny or Sobel.

B. Hough Transform One of the best classic methods to detect circles is the

circle Hough transform [14] which is based on curve fitting and uses the circle formula to detect it. Some modifications have been proposed to increase the detection rate or reduce its computational complexity.

This technique can be used for representing objects besides lines. For this case, a circle can be parameterized as:

222 )()( Rbyax =−+− (2)

Here, (a, b) is the coordinate of the center of the circle that passes through (x,y), and R is its radius. Since there are three parameters for this equation, it follows that the Hough transform will be a three-dimensional image. Therefore circles require more computation to find than lines. For this reason the Hough transform is more typically used for simpler curves.

The proposed Hough transform method uses a 2D accumulator array to detect circle centers and the radius is kept fixed to minimize the computation.

The strength of the Hough transform is that it is not very sensitive to imperfect data or noise, this makes it very robust. The Hough transform even manages to detect objects that are overlapping or semi-occluded.

IV. IDENTIFICATION OF SIGNS In the second part of algorithm, the classifier decides if the

acquired image represents a speed sign and in that case it also decides which speed it imposes.

The classifier has two main parts: preprocessing and correlation (figure 2). The correlation part uses reference images created from a training set in the pre-acquired database.

Figure 2. Basic steps in a traffic sign recognition algorithm

A. Image Pre-processing Before its classification, sign is passed through a number

of preprocessing steps. Initially, panel delivered from (ROI) detector is cut out of the current image .This ROI is resized to 80 by 80 in order to minimize the number of templates in database for the correlation step. A median filter is then applied to reduce noise and redundant information before binarization stage. The output obtained is just the number in panel and some other negligible information.

The last step in preprocessing is how to keep just the number of interest, because we don't need to keep the zero number which is redundant information in most speed panel.

To solve this problem, several methods have been used such as the k-means algorithm and label connected. In this design, we choose the simple and efficient method which is the label connected. This approach needs some parameters such as connectivity to define which pixels are connected to each other. However, the disadvantage of this method is that we need to repeat this operation for many times.

B. Correlation In parallel with the label connected method, the correlation

is used to compare the last output image to the template saved in the database in order to identify the right speed and pass this information for the driver to take a decision.

In fact, the cross correlation allows to determine the degree of similarity between two similar images. After applying this task, biggest percentage will guarantee a best identification.

V. EXPERIMENTS AND RESULTS

A. High level validation and test The presented above algorithms have been tested and

validated on a set of video sequences captured from different road and motorway in different weather and time condition.

All experiments were done on a Core2Duo Laptop, with 1.83 GHZ, 1Go RAM under Matlab 7.3 and Borland Builder 6.0 environment.

Figure 3 describes the different stages of traffic signs detection. The obtained image shows a satisfactory result.

Figure 3. Traffic sign detection

However, false detections have been obtained especially during weather change and important illuminations. This failure is more crucial when using RGB color space which is more sensitive to lighting conditions (fog, snow, heavy rain...) than other space such as HSV or YCbCr color spaces.

Computation times of the different tasks of traffic sign detection are given in Table 1. It shows that Hough Transform is the most expensive in terms of rapidity. Thus, to improve this penalized task we have chose to operate with a fixed radius.

TABLE I. COMPUTATION TIME EVALUATION

Task Computation time (sec) Percentage

Average Filter 0.7666 23.79 % Red Segmentation 0.0569 1.76 % Edge Detection 0.0752 2.33 %

Hough Transform 2.3234 72.10 %

B. Hardware implementation The system is designed around a Cyclone® II 2C35 FPGA

manufactured by Altera which connected to 8-MB SDRAM, 512-KB SRAM and 4-MB Flash. The DE2 board includes a sufficient number of robust interfaces such as video-in TV

Decoder, VGA (10-bit DAC), 10/100 Ethernet, an infrared (IrDA) port.

For experiments that require a processor and simple I/O interfaces, it is easy to instantiate Altera’s Nios II processor and use interface standards such as RS-232 and PS/2.

Our design is based on the use of NIOS II softcore processor (figure 4). When the FPGA device on the DE2 development board is configured with the Quartus II project encapsulating nios2_quartus2_project, the external physical pins on the FPGA are used by the design to connect to other hardware on the Nios development board, allowing the Nios II embedded processor to interface with RAM, flash memory, LEDs, LCDs, switches, and buttons.

Figure 4. Nios II based architecture

A set of development tools is provided by Altera to support all stages of the design process for logic circuits, including design entry, synthesis, placement and routing, simulation, and device programming. The DE2 Control Panel tool, which runs on PC platforms under Windows XP, allows remote control of the DE2 board through a USB cable (figure 5).

Figure 5. The DE2 Control Panel concept

Once the Nios II softcore processor is instantiated and downloaded in FPGA device, QVGA color image (320x240) is sent to SRAM memory by means of the control panel tool. All tasks of traffic sign detection are programmed in AINSI C, executed by the Nios II processor and evaluated with the DE2 Control panel.

The Nios II processor is created using the SOPC Builder wizard. Within this wizard, we can specify the settings for the Nios II processor, add peripherals, and select the bus connections, I/O memory mapping, and IRQ assignments for the processor.

The Nios® II integrated development environment (IDE) is the software development tool for the Nios II family of embedded processors. All software development tasks can be accomplished within the Nios II IDE, including editing, building, and debugging programs.

Once compiled, the C code is executed by Nios II and tested on QVGA image stored yet in SRAM memory. The computation time of each executed task is presented in table 2.

TABLE II. COMPUTATION TIME

Task Computation Time (s)

Average Filter 1,658 Red Segmentation 0,197 Edge Detection 0,386 Hough Transform 15,093

Results in table 2 show clearly how the Hough Transform technique spend a big time computation, since it contains non linear operations as the calculation of a square root which are very consumer on time. At the same time in spite of the choice of fixed radius for a 2D accumulator array, Hough method has remained much consumer on memory and time and needs to be more optimized to satisfy real-time detection.

VI. CONCLUSION AND FUTURE WORK In this paper, a traffic sign detection and recognition

algorithm has been studied and optimized. The proposed approach has presented high performance since we use at the same time color and shape features.

As well, some improvements can be provided to reduce sensitivity to lighting variation by using an appropriate color space.

In future work, we aim to bring further improvement in computational speed of some steps of treatment by using the new Hardware Acceleration Compiler (C2H). As its name implies, C2H tool allows C routines to be plucked from the normal Nios II software flow and compiled into high-performance hardware. In the same way, a hardware/software co-design can decrease significantly computation time.

REFERENCES [1] Richard Bishop, “Intelligent Vehicle Technology And Trends”, © 2005

Artech House Intelligent Transportation Systems Library, ISBN 1-58053-911-4.

[2] Vavilin Andrey and Kang Hyun Jo, “Automatic Detection and Recognition of Traffic Signs using Geometric Structure Analysis”,

SICE-ICASE International Joint Conference, October 2006, Bexco, Busan – Korea.

[3] De la Escalera A., Armingol J.M. and Mata M., “Traffic sign recognition and analysis for intelligent vehicles”, Image and Vision Computing, 21:247-258, 2003.

[4] Miura J., Kanda T. and Shirai Y., “An active vision for real-time traffic sign recognition”, Proc. IEEE Conf. on Intelligent Transportation Systems, pages 52-57, Dearbom, MI, 2000.

[5] Torresen J., Bakke J.W., and Sekania L., “Efficient recognition of speed limit signs”, Proc. IEEE Conf. on Intelligent Transportation Systems (ITS), Washington DC, 2004.

[6] D. Judd, D. MacAdam, G. Wyszecki, Spectral distribution of typical daylight as a function of correlated color temperature, J. Opt. Soc. Am. 54 (8) (1964) 1031–1040.

[7] Gavrila D.M., “Traffic sign recognition revisited”, Proc of 21st DAGM symposium fur Musterekennung, pp. 86-93, Springer-Verlag, 1999.

[8] Barnes N. and Zelinsky A., “Real-time radial symmetry for speed sign detection”, Proc. IEEE Intelligent Vehicle Symposium, pages 566-571, Parma, Italy, 2004.

[9] García-Garrido M. A., Sotelo M. A., and Martín-Gorostiza E., “Fast traffic sign detection and recognition under changing lighting conditions”, Proc. IEEE Intelligent Transportation Systems conference, pages 811-816, Toronto, Canada, 2006.

[10] A. de la Escalera and L. Moreno. Road traffic sign detection and classification. IEEE Trans. Indust. Electronics, 44:848–859, 1997.

[11] D. M. Gavrila. Traffic sign recognition revisited. In Mustererkennung (DAGM), Bonn, Germany, 1999. Springer Verlag.

[12] P. Paclik, J. Novovicova, P. Somol, and P. Pudil. Road sign classification using Laplace kernel classifier. Pattern Recognition Lett., 21(13–14):1165–1173, 2000.

[13] Mohamed Bénallal, Jean Meunier, "Real-time color segmentation of road signs ", Montréal, May/mai 2003.

[14] R.C. Gonzaltz, R.E.Woods, Digital image processing, Addision-Wesley, New York, I992