1 Automobile Lane Detection System- on-Chip Integrated with Mixed Signal Mode CMOS Image Sensor IEEE 9th International Symposium on Consumer Electronics

1

Automobile Lane Detection System-on-Chip Integrated with Mixed

Signal Mode CMOS Image Sensor

IEEE 9th International Symposium on Consumer Electronics (ISCE 2005)

Pei-Yung Hsiao¹, Hsien-Chein Cheng¹, Chun-Wei Yeh¹, Shih-Shinh Huang², and Li-Chen Fu²

¹ Department of Electronic Engineering Chang Gung University

² Department of Electrical Engineering National Taiwan University

2

Chang Gung University & National Taiwan University

OUTLINE

The Problem & Motive Brief Introduction to Algorithms Architecture and Circuits description Simulation and Results Conclusion

3


Original Idea for the Problem

Capture Image

Land map

We are interested in developing a System-on-Chip, Soc, which can capture image as well as produce vehicle lane map at the same time.

4


Motive The areas of Intelligent Transportation

System, ITS, include lane detection, obstacle recognition, vehicle detection, car following, etc.

Our goal in this investigation is to develop a CMOS imager to achieve real-time image capture and lane detection, simultaneously, for intelligent automotive driver awareness/assistance system.

5


Automotive IC Design

Vehicle Detection & Tracking

Driver Assistance System

Lane Departure Prevention

Automobile Lane Detection SoC

Built in

The target chip can be defined as a component device for intelligent vehicles.

6


Widespread Applications

The proposed imager without demanding extra ADC circuits for signal transformation is a single low-cost and compact chip for used in the thousands of consumer electronics not limited to ITS.

7


Introduction(1/4) From the referenced literatures, there are a lot

of vision-based lane detection algorithms proposed in recent 10 years [1-6] (1995-2004).

In 1995, Kluge and Lakshmanan [3] proposed the LOIS (Likelihood of Image Shape) lane detection, which is able to detect lanes even in situations with shadows or broken lanes by using a stochastic optimization procedure.

8


Introduction (2/4) In 1995, Broggi [5] proposed an edge-based ro

ad detection algorithm, while it is effective only for the well-painted road.

In 1999, Takahashi, etc. [4] divided the parameter space of the lane model to generate the lane marking patterns and then applied the voting scheme to find the lane boundary.

9


Introduction (3/4) In 2004, Huang, etc [1] proposed an on-board

vision system for lane recognition and front-vehicle detection to enhance driver's awareness.

Regarding to high recognition rate and hard-wired regularity, we adopted Peak-Finding based lane detection algorithm from Huang, etc [1], which has high recognition rate about 96%.

10


Introduction (4/4) According to Huang’s algorithm, we also

developed an auto-regulated threshold circuit to automatically adjust the threshold for the lane detector to adapted to different weather conditions.

11


Architecture and Circuit description Our chip can be divided into three parts, such

as analogue capturing and processing circuits, digital processing circuits and digital control unit.

The analogous circuits include 2-D pixel cell array, CDS module, 1-D Gaussian filter and Peak-Finding module.

The digital processing circuits are composed of Line Point Allocation module, column selector and row selector.

12


SOC for real-time image capture and lane detection

CMOS Image Sensor

Array

Gaussian Filter Module

Peak-Finding Module

Control

Unit

Lane-Point Finding Module

Row Selector

Lane-Point Output

Analogous Processing circuits

Digital Processing Circuits

CDS Circuit

Upper Region

Column

Selector

13


Pixel Cell & Sensor Array The developed sensor array consists of two types of p

ixel cells. Our sensor array prototype is made up of 64*64 effecti

ve pixels. The upper region containing 16*64 pixels is ignored in

back-end processing to promote the computing efficiency.

The other regions are horizontally partitioned into three sub-regions. Each sub-region consists of 16 rows.

The 12th row in the sub-region or in the upper region is defined as a 1-D sample array. Consequently, we have four 1-D sample arrays in our sensor array.

14


Reset

Photodiode

SEL

IpIpd

X

Reset

Photodiode

SEL

IpIpd

X

Ips

1.10 i-2,10 i-1,10 i,10 i+1,10 i+2,10Row Selector

1,11 i-2,11 i-1,11 i,11 i+1,11 i+2,11

1,12 i-2,12 i-1,12 i,12 i+1,12 i+2,12

1,64 i-2,64 i-1,64 i,64 i+1,64 i+2,64

CDS Circuit

Column SelectorCLK

Gaussian Filter (Current)

Gaussian Filter (Preivois)

IG(i,j)

IG(i-1,j)

Ips

Ip(i-1,j)Ip(1,j) Ip(i-2,j) Ip(i,j) Ip(i+1,j)

Ip(i+2,j)

64,10

64,11

64,12

64,64

Ip(64,j)

1,1 i-2,1 i-1,1 i,1 i+1,1 i+2,1 64,1

Normal cell

Sampling cell

15


Dual 1-D Gaussian Filters Each 1-D Gaussian filter includes 64 Gaussian

mask cells and a current divider. Each Gaussian mask cell consists of 3 current mirrors in 7 transistors and two OR gates.

The Gaussian filter module is used for smoothing the selected pixel by referring to a couple of right and left neighbors to eliminate noisy points in the original image.

16


Current Gaussian Filter

Previous Gaussian Filter

IG(i,j)

IG(i-1,j)

Ip(1,j) Ip(i,j) Ip(64,j)

S(1)

S(3)

S(2)

S(i)

S(i+

1)

S(i+

2)

S(i-2

)

S(i-1

)

S(64

)

S(63

)

S(62

)

S(1)

S(2)

S(63

)

S(62

)

S(61

)

S(60

)

S(i-1

)

S(i)

S(i+1

)

S(i-3

)

S(i-2

)

Dual 1-D Gaussian Filters

17


Peak-Finding Module (1/3) The 1st part of the Peak-Finding Module can a

ccumulate and average current, Iavg, from the aforementioned sample arrays, Ips.

The averaged current from sample array, Iavg, was generated according to the following equation.

),*1612(1 64,1

1,0

jiIpsn

In

ji

avg

18


IG(i,j)IG(i-2,j)

Auto-Regulated Threshold Circuit

Vrefn

Iavg

M1 M2

Isth

Threshold Mapping Circuit

Pp(i,j)





Vref

ITH

Vbias Ibias

Iavg

From Sample Array

Vcn

Ips

Irefn

P1 P2 P3 P4 P5 P6 P7 P8 P9

P73 P72 P71 P(i-1,j) P(I,j) P68 P67 P66 P65P101 P100

P129 P130 P131 P132 P133 P134 P135 P136 P137P102 P103

P38 P37

P35 P36

Pp(i,j)

Lb(i,j)Lane-Point Output

Peak-Point Output

Peak-Finding Module

Line-Point Allocation

19


Peak-Finding Module (2/3) The 2nd part of PFM is called as auto-regul

ated threshold circuit. It compares average current, Iavg, with four preconfigured currents, Irefn, and then produces the threshold current, Isth.

The total threshold current, ITH can be noted by the following equation.

, where

4

1n

sthbiasTH III

20 )(

2

2TCnn

M

Msth VV

LWI cu

20


Peak-Finding Module (3/3) Inside the auto-regulated threshold circuit, each

threshold mapping circuit control a threshold current. It can be noted by the following equation:

According to the 3rd part of the PFM, If the current pixel is a peak point, the output value will be 1, otherwise, it should be 0.

otherwise

jiIIjiIifjiP

GTHG

p

,0),1(),(,1

),(

otherwise

IIifV

avgrefn

Cn

,0,1

20 )( 5

1

1

1Tref

m

nn

M

Mrefn VV

R

RLWI

m

cu

, where

21


Line-Point Allocation The Lane-Point Allocation Module expressed a

s the following equation is composed of two digital functions, such as the line segment filter and the lane point selector.

Lane points, Lb(i,j), are obtained, and represented by only one pixel width in each row.

])1,(1,[,,1,3

3

3

3

mn

jmiPjniPjiPjiPjiL ppppb

22


1 2 64i (Col)

1 1

1 2 64

2 2

1 2 64

63 63

1 2

64 64

64 1 2

1

1 642 4097 4098

1,1 1,164,64Peak-Point

Pp(i,j)Lane-Boun Lb(i,j)

63,641,6464,6363,631,6364,6263,21,264,163,1---

--- --- 64,163,162,1 62,2 64,6263,62 64,6363,6362,63 64,6463,6462,64

Timing Cycle 65 66 128 3969 3970 4032 4033 4044 4096

i (Row)

clk R

clk C

Reset

The clock frequency of the Row Selector (clk R) is 64 times of the Column Selector (clk C). In this case, the frequency of the Column Selector is 25MHz and the frequency of the Row Selector is 0.78MHz.

Timing Diagram

23


HSPICE Simulation (1/2)

(a)(b)(c)

Peak-Point

(a) is the output current of the current Gaussian Filter. (b) is the output current of the previous Gaussian Filter. (c) is the results of the Peak-Finding Module.

1.5us

24


HSPICE Simulation (2/2)(a)

(b)

(c)

(d)

(a) is the simulation results of the current Gaussian Filter. (b) is the simulation results of the previous Gaussian Filter. (c) is the simulation result of the Peak-Finding Module. (d) is the simulation results of the Lane-Point Allocation Module.

5us 25us

25


Software Simulation in C

(a) Original image (512x512 > 320x240[1])

(b) Lane map points generated by Peak-Finding algorithm.

(c) Original image (64x64)

(d) Lane map points generated by Peak-Finding algorithm.

From [1], noises are to be removed by the followed post processing.

26


Experimental Results

(a) Original image in 32 * 32

(c) Result generated by our chip

(b) Result generated by software

27


Chip Layout in 64 * 64

28


Specification of The Proposed CMOS Imager

Item ValuesPixel Count 64(H) X 64(V)Pixel Size 18.45 um(H) X 21.8 um (V)Aperture Size 12.45 um(H) X 9.6 um (V)Fill Factor 29.7 %Image Size 1217.7 um (H) X 1455.05 um

(V)Chip Size 2191.4 um (H) X 2389.8 um

(V)Operation Clock 25MHzOperation Voltage 3.3 VPower consumption 159.4mW

29


Conclusion Our investigation includes a 2-D image sensor

array embedded with four modularized circuits: ----- four 1-D sample array by different pixel cell

design for accumulating the sampled currents; ----- dual 1-D Gaussian filers coupling as an

analogue image smoothing module; ----- an analogue design for Peak-Finding function

associating with a novel auto-regulated threshold operation;

----- a sophisticated digital implementation for Lane-Point allocation.

30


Conclusion, cont. A new current-mode mixed signal

design of CMOS image sensor integrated with Peak-Finding based lane detection algorithm is developed.

The proposed low-cost and one compact chip solution can grab the road images from the real world and successfully detect the lane markers simultaneously, in real time.

31


Reference (1/4)[1] Shih-Shinh Huang, Chung-Jen Chen, Pei-Yung Hsiao, and Li-

Chen Fu, “On-Board Vision System for Lane Recognition and Front-Vehicle Detection to Enhance Driver's Awareness”, IEEE International Conference on Robotics and Automation, vol. 3, 26 April – 1 May, 2004, pp.2456–2461.

[2] Yue Wang, Eam Khwang Teoh and Dinggang She, “ Lane detection using B-snake”, , International Conference on Information Intelligence and Systems, 31 Oct. - 3 Nov., 1999, pp.438 – 443.

[3] Kluge, K., and S. Lakshmanan,, “A Deformable-Template Approach to Lane Detection”, in I. Masaky, editor, Proceedings IEEE Intelligent Vehicle’95, Detroit, 25-26 Sept., 1995, pp.54-59.

[4] Takahashi, A., Ninomiya, Y., Ohta, M., and Tange, K., “A Robust Lane Detection Using Real-Time Voting Processor”, IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems, 5-8 Oct., 1999, pp.577–580.

32


Reference (2/4)[5] A. Broggi, “Robust Real-Time Lane and Road Detection in

Critical Shadow Conditions,” Computer Vision, 1995, Proceedings., International Symposium on, 21-23 Nov. 1995 pp.353-358

[6] Li, Q., Zheng, N., and Cheng, H.,“ Springrobot: A Prototype Autonomous Vehicle and Its Algorithms for Lane Detection”, IEEE Transactions on Intelligent Transportation System, vol. 5, no. 4, Dec., 2004, pp.300-308.

[7] Coulombe, J., Sawan, M. and Wang, C., “Variable Resolution CMOS Current Mode Active Sensor”, IEEE International Symposium on Circuits and Systems, vol. 2, 28-31 May, 2000, pp.293 – 296.

[8] Tabet, M., Hornsey, R.,“ CMOS Image Sensor Camera with Focal Plane Edge Detection”, Canadian Conference on Electrical and Computer Engineering, vol. 2, 13-16 May, 2001, pp.1129 – 1133.

33


Reference (3/4)[9] Pei-Yung Hsiao, Yu-Chun Hsu, Wen-Ta Lee, Chia-Chun Tsai,

and Chia-Hao Lee, ”An Embedded Analog Spatial Filter Design of The Current-Mode CMOS Image Sensor”, IEEE Transactions on Consumer Electronics, vol. 50, no. 3, Aug., 2004, pp.945–951.

[10] N. Yang and G. Jianhong, “ A 256x256 Pixel Smart CMOS Image Sensor for Line Based Stereo Vision Applications”, IEEE Journal of solid state circuits, vol. 35, no. 7, July, 2000, pp.1055-1061

[11] Yuan, J. and Svensson, C., “High-speed CMOS circuit technique,” IEEE J. Solid-state Circuits, vol. 24, no. 2, 1989, pp. 62-70.

[12] Byungsoo Chang, Joonbae Park and Wonchan Kim,“A 1.2 GHz CMOS dual-modulus prescaler using new dynamic D-type flip-flops”, IEEE Journal of Solid-State Circuits, vol. 31, no. 5, May, 1996. pp.749-752

34


Reference (4/4)[13] Fish, A and Yadid-Pecht, o.,”CMOS current/voltage mode

winner-take-all circuit with spatial filtering”, The IEEE International Symposium on Circuits and Systems, vol. 3, 6-9 May, 2001, pp.636-639.

[14] Nakamura, J., Pain, B., Nomoto, T., Nakamura, T. and Fossum, E.R., “On-focal-plane signal processing for current-mode active pixel sensors”, IEEE Transactions on Electron Devices, vol. 44, no. 10, Oct., 1997, pp.1747 – 1758

Documents

1 Automobile Lane Detection System- on-Chip Integrated with Mixed Signal Mode CMOS Image Sensor IEEE 9th International Symposium on Consumer Electronics