F2008-08-099 LIGHT STRIPE PROJECTION-BASED PEDESTRIAN DETECTION DURING AUTOMATIC PARKING OPERATION 1Jung, Ho Gi*, 1Kim, Dong Suk, 1Kang, Hyoung Jin, 2Kim, Jaihie 1MANDO Corporation, Republic of Korea, 2Yonsei University, Republic of Korea KEYWORDS – Driver assistant system, automatic parking assistance system, collision warning, pedestrian detection, light stripe projection ABSTRACT – Automatic parking system is one of the most interesting driver assistance systems because it is recognized as a convenience system used in daily life and it is expected to become the first widely deployed driver assistance system using state-of-the-art technologies. Generally, automatic parking system consists of three components: target position designation, path planning, and path tracking by active steering. There are various methodologies to establish the target parking position. We proposed light stripe projection (LSP)-based target position designation to deal with parking situation with dark illumination condition such as in underground parking space (1). Although responsibility during parking operation is on the driver, detection of pedestrian during backing manoeuvre should be beneficial to the driver. The authors of (2) developed vision-based pedestrian detection using moving obstacle detection. However, it assumed that visual condition was good such that pedestrian had distinctive appearance in captured image. This paper proposes that LSP-based 3D reconstruction technology used in target position designation can be used for pedestrian recognition in dark illumination condition. TECHICAL PAPER – INTRODUCTION As shown in Terry Costlow’s automotive camera needs prediction (3), vision-based parking assistance system attracts rapidly rising interest. As shown in Fig. 1, the number of parking assistance system is expected to grow and reach 23,353k units/year in 2013. Particularly, automatic parking system is one of the most interesting driver assistance systems because it is recognized as a convenience system used in daily life and it is expected to become the first widely deployed driver assistance system using state-of-the-art technologies.

Fig. 1. Needs prediction of automotive camera (3).

Generally, automatic parking system consists of three components: target position designation, path planning, and path tracking by active steering. There are various methodologies to establish the target parking position (4). We proposed light stripe projection (LSP)-based target position designation to deal with parking situation with dark illumination condition such as in underground parking space (1). The system adds one light plane projector, that is line laser module, to rear view camera-based automatic parking system. The projected light plane makes light stripe feature (LSF) on objects, whose image captured by camera can be transformed into three-dimensional (3D) information. With reconstructed 3D information, the system recognizes clusters and finds free parking space. The system was expected to be an economic solution for dark illumination condition, which was one of the most difficult problems to vision-based approaches. Two kinds of accidents are considerable during parking operation: crash with adjacent vehicles or pillars, and crash with passing pedestrian. The first accident type is supposed to be caused by inexact target position designation or poor tracking performance. They can be improved and guaranteed before the deployment. Contrarily, the second accident type can not be predicted because passing pedestrian is not the part of automatic parking system. Therefore, although responsibility during parking operation is on the driver, detection of pedestrian during backing manoeuvre should be researched. The authors of (2) developed vision-based pedestrian detection using moving obstacle detection. However, it assumed that visual condition was good such that pedestrian had distinctive appearance in captured image. This paper proposes that LSP-based 3D reconstruction technology used in target position designation can be used for pedestrian recognition in dark illumination condition. Target parking position is established by LSP-based free space detection method (1) then is updated during backing manoeuvre by odometry. During parking operation, system continuously detects LSF and check whether any LSF cluster in bird’s eye view is located inside the target parking position. LSP-BASED 3D RECONSTRUCTION System Configuration Proposed system can be implemented simply by installing a light plane projector at the backend of vehicle as shown in Fig. 2. NIR (Near InfraRed) line laser is used as the light plane projector not to border neighboring people. Because parking assist system should provide backward image to driver, band-pass filter, commonly used with NIR light source, can not be used. On the contrary, in order to acquire image in NIR range, infrared cut filter generally attached to camera lens is removed. To capture backward image with large FOV (Field Of View), fisheye lens is used. With radial distortion parameters measured through calibration procedure, input image is rectified to be undistorted image. In order to robustly detect light stripe without band-pass filter, difference image between image with light projector on and image with light projector off is used. Turning the light plane projector on during short period, system acquires backward image including light stripe. Turning the light plane projector off, system acquires visible band image. Difference image between them can extract light stripe irrespective of surrounding environment. Furthermore, because light plane projector is turned on during only short time, it can guarantee eye-safe operation with comparatively high power light source (5).

Fig. 2. System configuration Light Stripe Projection Method Finding the intersecting point between plane Π and line Op can calculate the coordinates of

stripe pixel point P as sown in Fig. 3. Π denotes the plane generated by light plane projector. Line laser projected onto object surface forms a light stripe. p (x, y) denotes the point on image plane corresponding to a point P (X,Y,Z) on light stripe.

Fig. 3. Relation between light plane and light stripe image.

The coordinate of P can be calculated as follows (6):

( )tan cos

tan sin cosx bX

f x yα ρ

α ρ ρ⋅ ⋅ ⋅

=− ⋅ + ⋅

( )tan cos

tan sin cosy bY

f x yα ρ

α ρ ρ⋅ ⋅ ⋅

=− ⋅ + ⋅

( )tan cos

tan sin cosf bZ

f x yα ρ

α ρ ρ⋅ ⋅ ⋅

=− ⋅ + ⋅

Light plane Π meets Y-axis at point P0 (0, -b, 0). The angle between the plane and Y-axis is α

and the angle between the plane and X-axis is ρ. Distance between camera and P0, i.e.

baseline, b and the between-angle α are calculated by configuration normalization. Fig. 4 shows the example of reconstructed 3D information.

Fig. 4. 3D reconstruction result. PARKING SPACE DETECTION AND TRACKING Viable free parking space is detected by four steps: occlusion detection, pivot detection, opposite side reference point detection and target parking position establishment (1). Once the target parking position is established, the system generates path plan and controls steering angle to track it (7). Occlusion Detection Occlusion is defined as a 3D point pair, which is located in adjacent column in stripe image and is far more than a certain threshold, e.g. ego-vehicle’s length, from each other in XZ plane. Occlusions are detected by checking the above mentioned conditions on detected light stripe. Two 3D points belonging to an occlusion have a property about whether it is left-end or right-end of consecutive stripe pixels. Here, left-end occlusion point and right-end occlusion point make one occlusion. If the left-end occlusion point of an occlusion is nearer than the right-end occlusion point from camera, the pair is supposed to be on the left-end of interesting object. In this application, the interesting object means object existing on the left

or right side of free space. If the left-end occlusion point is further than the right-end occlusion point from camera, the occlusion is supposed to be on the right-end of interesting object. Such characteristic is attached to occlusion as directional information. In Fig. 5(a), the occlusion at the left-end of interesting object is drawn by red line and the occlusion at the right-end of interesting object is drawn by green line. Pivot Detection Pivot occlusion is defined as an occlusion which satisfies the below conditions: 1) There is free space in the direction of occlusion. The region of free space checking is semicircle, of which straight side is the line between occlusion points and of which radius is ego-vehicle’s width and of which center is the nearer occlusion point. 2) It is far from FOV border by sufficiently large distance, e.g. ego-vehicle’s width. 3) It is nearer than any other candidates from optical axis. Pivot, center of rotation, is the nearer occlusion point of pivot occlusion. Fig. 5(b) shows recognized pivot with ‘+’ marking. Recognition of Opposite-Side Reference Point It is assumed that recognized 3D points, whose distance from pivot is smaller than a certain threshold, e.g. ego-vehicle’s length, in the direction of pivot occlusion, belong to opposite-side object. Fig. 5(c) shows detected 3D points belonging to opposite-side object. Among these 3D points, one point nearest to pivot becomes the initial opposite-side reference point. Using points whose distance from opposite-side reference point is smaller than a certain threshold, e.g. 2/3 of ego-vehicle’s length, in the direction going away from camera and perpendicular to pivot, the direction of opposite-side object’s side can be estimated. Fig. 5(d) shows the estimated direction of opposite-side object’s side based on 3D points marked by yellow color. Estimated side of opposite-side object is marked by blue line. Establishment of Target Parking Position Rectangular target position is established at the center between pivot and opposite-side reference point. Target position’s short side is aligned to the line between pivot and opposite-side reference point. The width of target position is the width of ego-vehicle and the length of target position is the length of ego-vehicle. Fig. 5(e) shows the finally established target position. Fig.5(f) shows the established target position projected on input image. Path Plan and Tracking Path plan can be generated by connecting three path sections: two straight line-segments and one circular arc. The first part is a straight line-segment passing the vehicle center in the longitudinal direction and the third part is the central line of target parking position. A circular arc connects the two line segments with mechanically allowable radius (7). To make vehicle model, vehicle data was used such as wheelbase, track, length, overhang and maximum steering angle. The vehicle model is popular Ackerman (or bicycle) model, assuming that no slip of wheels occurred due to low speed. With the vehicle model, the system can update the position of the subjective vehicle. By applying the same motion to the target position inversely, the system can update the target position continuously (7).

(a) Detected occlusion points and occlusion (b) Pivot detection result

(c) Opposite-side reference points (d) estimated direction of opposite-side object’s side

(e) Established target potion (f) Target position projected on input image.

Fig. 5. Free parking space detection procedure.

LSP-BASED PEDESTRIAN DETECTION Target parking position is established by LSP-based free space detection method then is updated during backing manoeuvre by odometry (7). During parking operation, system continuously detects LSF and check whether any LSF cluster in bird’s eye view is located inside the target parking position. Fig. 6(a) shows detected LSFs when a pedestrian is moving inside the target parking position. Fig. 6(b) shows LSFs in bird’s eye view and recognized pedestrian appearing as a LSF cluster inside the target parking position.

(a) Rear view scene with detected light stripe features. Circle denotes a pedestrian inside the target parking position.

(b) Detected light stripe features in top view. The pedestrian denoted by circle causes warning to the driver

because it is inside the target parking position.

Fig. 6. Light stripe projection-based target parking position establishment and pedestrian detection.

Target parking position Planned path

Detected pedestrian

EXPERIMENTAL RESULTS To verify the efficiency of the proposed method, situations with pedestrian at different distances were investigated. Fig. 7 shows the situations and recognized pedestrians. Fig. 7(a) shows the situation when a pedestrian is entering the adjacent vehicle and is inside the target parking position. Fig. 7(b) and (c) shows the situations when a pedestrian is on the backing path. It is noteworthy that the same method can be applied to pedestrians with various distances.

(a) When a pedestrian is entering the adjacent vehicle.

(b) When a pedestrian is on the backing path.

(c) When a pedestrian is just behind the back of the subjective vehicle.

Fig. 7. Detected pedestrian at various distances.

CONCLUSION Our previous paper (7) proposed a novel light stripe projection based free parking space recognition method in order to overcome the common drawbacks of existing vision based target position designation methods in dark indoor parking site. The proposed method is expected to be practical solution because it can be implemented simply by installing a low-cost light plane projector on existing parking monitoring system and it uses simple mathematics and computation to recognized 3D information of parking site. Various experiments show that the proposed method can successfully recognize target parking position in spite of various illumination conditions. In this paper, we proposed that LSP-based 3D reconstruction method can not only establish target parking position but also detect passing pedestrian to avoid accident in dark illumination condition. As the system updates target parking position using odometry and acquires 3D information continuously, it can detect pedestrian simply by checking whether there is range clusters inside the target parking position. REFERENCES (1) Ho Gi Jung, Dong Suk Kim, Pal Joo Yoon, and Jaihie Kim, “Light Stripe Projection

based Parking Space Detection for Intelligent Parking Assist System”, Proceedings of the 2007 IEEE Intelligent Vehicle Symposium, Jun. 13-15, 2007.

(2) S. Wybo, R. Bendahan, S. Bougnoux, C. Vestri, F. Abad, and T. Kakinami, “Improving backing-up manoeuvre safety with vision-based movement detection”, IET Intelligent Transport Systems, vol. 1, no. 2, Jun. 2007, pp. 150-158.

(3) Terry Costlow, “Shifting into active mode”, Automotive Engineering International, Jun. 2007, pp. 44-46.

(4) Ho Gi Jung, Young Ha Cho, Pal Joo Yoon, and Jaihie Kim, “Scanning Laser Radar-Based Target Parking Position Designation Method for Parking Aid System”, IEEE Transactions on Intelligent Transportation Systems, Accepted for future publication, Digital Object Identifier: 10.1109/TITS.2008.922980.

(5) C. Mertz, J. Kozar, J. R. Miller, and C. Thorpe, “Eye-safe laser striper for outside use,” in Proc. IEEE Intelligent Vehicle Symposium, Jun. 17-21, 2002, pp. 507-512, Vol. 2.

(6) Reinhard Klette, Karsten Schlüns, and Andreas Koschan, Computer Vision – Three Dimensional Data from Images, 1998, Springer-Verlag.

(7) Chi Gun Choi, Dong Suk Kim, Ho Gi Jung, and Pal Joo Yoon, “Stereo Vision Based Parking Assist System”, SAE Paper No.: 2006-01-0571.