An Efficient and Robust Moving Shadow Removal Algorithm and Its

Hindawi Publishing CorporationEURASIP Journal on Advances in Signal ProcessingVolume 2010, Article ID 945130, 19 pagesdoi:10.1155/2010/945130

Research Article

An Efficient and Robust Moving Shadow Removal Algorithm andIts Applications in ITS

Chin-Teng Lin,1 Chien-Ting Yang,1 Yu-Wen Shou,2 and Tzu-Kuei Shen1

1Department of Electrical and Control Engineering, National Chiao Tung University, Hsinchu 300, Taiwan2Department of Computer and Communication Engineering, China University of Technology, Hsinchu 303, Taiwan

Correspondence should be addressed to Yu-Wen Shou, [email protected]

Received 1 December 2009; Accepted 10 May 2010

Academic Editor: Alan van Nevel

Copyright © 2010 Chin-Teng Lin et al. This is an open access article distributed under the Creative Commons Attribution License,which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

We propose an efficient algorithm for removing shadows of moving vehicles caused by non-uniform distributions of lightreflections in the daytime. This paper presents a brand-new and complete structure in feature combination as well as analysisfor orientating and labeling moving shadows so as to extract the defined objects in foregrounds more easily in each snapshot ofthe original files of videos which are acquired in the real traffic situations. Moreover, we make use of Gaussian Mixture Model(GMM) for background removal and detection of moving shadows in our tested images, and define two indices for characterizingnon-shadowed regions where one indicates the characteristics of lines and the other index can be characterized by the informationin gray scales of images which helps us to build a newly defined set of darkening ratios (modified darkening factors) based onGaussian models. To prove the effectiveness of our moving shadow algorithm, we carry it out with a practical application of trafficflow detection in ITS (Intelligent Transportation System)—vehicle counting. Our algorithm shows the faster processing speed,13.84 ms/frame, and can improve the accuracy rate in 4% ∼10% for our three tested videos in the experimental results of vehiclecounting.

1. Introduction

In recent years, the researches about intelligent videosurveillances have increased noticeably. Foreground objectdetection could be considered as one of the fundamentaland critical techniques in this field. In the conventionalmethods, background subtraction and temporal differencehave been widely used for foreground extraction in case ofusing stationary cameras. The continuing improvements inbackground modeling techniques have led to many new andfascinating applications like event detection, object behavioranalysis, suspicious object detection, traffic monitoring, andso forth. However, some factors like dynamic backgroundsand moving shadows might affect the results of fore-ground detection and make the problems more complicated.Dynamic backgrounds, one of the factors, might detect andtreat the escalators and swaying trees as the foregroundregions. Another factor, moving shadows, occurred whenlight was blocked by moving objects, usually misclassified theforeground regions. This paper would focus on the studies

of moving shadows and aim at developing an efficient androbust algorithm of moving shadow removal as well as therelated applications.

About the studies on influences of moving shadows,Zhang et al. [1] classified these techniques into four cat-egories, including color model, statistical model, texturalmodel, and geometric model. Color model used the differ-ence of colors between the shaded and nonshaded pixels.Cucchiara et al. [2] removed moving shadows by usingthe concept on HSV color space that the hue componentof shaded pixels would vary in a smaller range and thesaturation component would decrease more obviously. Someresearchers proposed shadow detection methods based onRGB color space and normalized-RGB color space. Yang etal. [3] described the ratio of intensities between a shadedpixel and its neighboring shaded pixel in the current image,and this intensity ratio was found to be close to that in thebackground image. They also made use of the slight changeof intensities on the normalized R and G channels betweenthe current and background image. Cavallaro et al. [4]

2 EURASIP Journal on Advances in Signal Processing

found that the color components would not change theirorders and the photometric invariant features would havea small variation while shadows occurred. Beside of colormodel, statistical model used the probabilistic functions todetermine whether or not a pixel belonged to the shadows.Zhang et al. [1] introduced an illumination invariance fea-ture and then analyzed and modeled shadows as a Chi-squaredistribution. They classified each moving pixel into theshadow or foreground object by performing a significancetest. Song and Tai [5] applied Gaussian model to representingthe constant RGB-color ratios, and determined whethera moving pixel belonged to the shadow or foregroundobject by setting ±1.5 standard deviation as a threshold.Martel-Brisson and Zaccarin [6] proposed GMSM (GaussianMixture Shadow Model) for shadow detection which wasintegrated into a background detection algorithm based onGMM. They tested if the mean of a distribution coulddescribe a shaded region, and if so, they would select thisdistribution to update the corresponding Gaussian mixtureshadow model.

The texture model assumed that the texture of theforeground object would be totally different from that of thebackground, and that the textures would be distributed uni-formly inside the shaded region. Joshi and Papanikolopoulos[7, 8] proposed an algorithm that could learn and detectthe shadows by using support vector machine (SVM). Theydefined four features of images, including intensity ratio,color distortion, edge magnitude distortion and edge gra-dient distortion. They introduced a cotraining architecturewhich could make two SVM classifiers help each other in thetraining process, and they should need a small set of labeledsamples on shadows before training SVM classifiers fordifferent video sequences. Leone et al. [9] presented a shadowdetection method by using Gabor features. MohammedIbrahim and Anupama [10] proposed their method byusing division image analysis and projection histogramanalysis. Image division operation was performed on thecurrent and reference frames to highlight the homogeneousproperty of shadows. They afterwards eliminated the leftpixels on the boundaries of shadows by using both column-and row-projection histogram analyses. Geometric modelattempted to remove shadowed regions or the shadowingeffect by observing the geometric information of objects.Hsieh et al. [11] used the histograms of vehicles and thecalculated center of lane to detect the lane markings, andalso developed a horizontal and vertical line-based methodto remove shadows by characteristics of those lane markings.This method might become ineffective in case of no lanemarkings.

As for some other approaches by combination of thementioned models, Benedek and Sziranyi [12] proposed amethod based upon LUV color model. They used “darkeningfactor”, the distortion of U and V channel, and microstruc-tural responses to be the determinative features, wheremicrostructural responses represented a local textured fea-ture. Their proposed algorithm integrated all those featuresby using Gaussian model and segmented foreground objects,backgrounds and shadows by calculating their probabilities.Xiao et al. [13] proposed a shadow removal method based

on edge information for traffic scenes which in sequenceconsisted of an edge extraction technique, the morphologicaloperations on removing the edges of shadows, and theanalysis of spatial properties for separating the occludedvehicles which resulted from the influences of shadows.They could reconstruct the size of each object and decidethe practical regions of shadows. However, this methodintrinsically could not deal with the textured shadowedregions like the regions with lane markings, and the morecomplicated cases of occlusions like the concave shapes ofvehicles would make this method fail to separate vehicles byuse of spatial properties.

The color information from color cameras might givefine results for shadow removal, but B/W (Black & White)cameras could provide the better resolution and moresensitive quality under lowly illuminating conditions. Thatwas the reason why B/W cameras rather than color cameraswould be much more popular for outdoor applications. Theshadow removal method based on color model might notwork in such situations. Similarly, texture model could havebetter results under the unstably illuminating conditionswithout the color information. However, texture modelmight give the poorest performances for the texturelessobjects. Geometric model could be more adaptive to thespecific scenes due to dependency upon the geometric rela-tions between objects and scenes, and would be prevailinglyapplied in simulated environments so far. Its biggest prob-lem in the heavily computational loading would obviouslyrestrict the related uses in real-time cases. By considering allthese methods, we proposed a fast moving shadow removalscheme by combining texture and statistical models. Ourproposed method was experimentally proved to be stable andused the texture model instead of color model to simplifyour systematic procedures efficiently. Furthermore, we madeuse of statistical methods to improve the performances ofsystems by successfully dealing with the textureless objects.This paper would be organized as follows, including theoverall architecture, foreground object extraction, featurecombination, experimental results on both our proposedmoving shadow removal algorithm and the additionalapplication (vehicle-counting), and conclusions.

2. Our Systematic Architecture forMoving Shadow Removal

In this section, we would introduce our entire architecturefor algorithms of moving shadow removal which consistedof five blocks, including foreground object extraction,foreground-pixel extraction by edge-based shadow removal,foreground-pixel extraction by gray level-based shadowremoval, feature combination, and the practical applications.The architecture diagram of our proposed algorithm wasshown in Figure 1.

3. Foreground-Object Extraction

As Figure 1 showed, the sequence of images in gray-levelshould be taken as the input of the foreground object

EURASIP Journal on Advances in Signal Processing 3

Gray level imagesequence

Foregroundobject

extraction

Gray level-basedshadow removalforeground pixel

extraction

Edge-basedshadow removalforeground pixel

extraction

Featurecombination

Trackingprocess

Result ofobject

detection

Figure 1: Architecture of the proposed shadow removal algorithm.

extraction processes, and the moving object with its min-imum bounding rectangle as the output of the algorithm.We would incorporate Gaussian Mixture Model (GMM)into our mechanism functioning as the development ofbackground image which is a representative approach tobackground subtraction. It would be more appropriate tochoose the way by background subtraction for extractingforeground objects rather than that by temporal differenceunder the consideration of all the pros and cons of thesetwo typical approaches. Furthermore, the latter could do abetter job in extracting all relevant pixels, and this paperaimed at tackling the problems generated from the trafficmonitoring systems where cameras would be usually setfixedly. Some previous studies [14, 15] have proposed astandard process of background construction, hence wewould put a higher premium on the following two parts,inclusive of foreground-pixel extraction by edge-based andgray-level based shadow removal.

3.1. Foreground-Pixel Extraction by Edge-Based ShadowRemoval. The main ideas for extracting foreground-pixels bythe information of edges in the detected object were inspiredby that the edges of object in interest would be identified

and removed much easier if the homogeneity in the shadowregion could be within a small range of variance. Relatively,we could also obtain the features of edges for nonshadowregions. The flowchart of foreground-pixel extraction byedge-based shadow removal was shown in Figure 2. Moreclearly, we firstly used Sobel operations to extract the edgesfor both GMM-based background images and foregroundobjects. Figures 3 and 4 showed the results of edge extractionby Sobel operations from BI edge and FO edgeMBR whereBI edge and FO edgeMBR represent the edges extracted frombackground images and foreground objects, respectively. Thesubscript “MBR” means the minimum bounding rectangle,the only region which we have to process inside.

In order to avoid extracting the undesired edges, forexample, the edges of lane marking or textures on the groundsurface, we technically took advantage of pixel-by-pixel max-operations on the extracted edges of background images andforeground objects, and the results would be expressed asMI edgeMBR in (1):

MI edgeMBR

(x, y

)

= max(

FO edgeMBR

(x, y

), BI edge

(x, y

)),

(1)

where (x, y) represents the coordinate of the pixel, and oneexample of MI edgeMBR would be illustrated in Figure 5.

Then, we subtracted BI edge from MI edgeMBR to obtainSt edgeMBR. Figure 6 showed the result of St edgeMBR. Sim-ilarly, Figure 7 showed the same result without using ourproposed procedures. It could be easily observed that theextracted edges of lane marking indeed reduced if ourproposed procedure could be applied appropriately. Figure 8indicated that our proposed procedure would also work wellon the images with textured roads.

To demonstrate the necessity of our proposed procedure,we circled the edges of lane markings and textures on groundsurface by red ellipses in both Figures 7 and 8(f). By using themax-operation, apparently, we could reduce the effect causedeither by lane markings or textures on ground surfaces andalso keep the homogeneous property inside the shadows.

After edge extracting, we used an adaptive binarizationmethod to obtain the binary image from St edgeMBR. Here,we took Sauvola’s method [16, 17], one kind of localbinarization methods, to provide good results even in thecondition of nonuniform luminance. Sauvola used an n ×n mask, covered on the image in each scanning-iteration,to calculate the local mean m(x, y) and standard deviations(x, y) of pixel-intensities in the mask as to determine aproper threshold according to the contrast in the localneighborhood of a pixel. If there is a high contrast in someregion of the image, the equation s(x, y) ≈ R may result inthe condition tfinal(x, y) ≈ m(x, y). To reduce the influencesby unimportant edges, we added a suppression term toSauvola’s equation. Equation (2) shows the revised equation.

tfinal(x, y

) = m(x, y

)[

1 + k

(s(x, y

)

R− 1

)]

+ Thsuppress,

(2)


Edge-based shadow removalforeground pixel extraction

Sobeloperation

Pixel by pixelmaximization

Sobeloperation

Subtraction

Boundaryelimination

Adaptivebinarization

Edge-basedshadow removal

foreground pixels

GMMbackground

Foregroundobject

Figure 2: The flowchart of foreground-pixel extraction by edge-based shadow removal.

(a) Background image (b) BI edge

Figure 3: Sobel edge extraction from background images.

(a) Foreground Object (b) FO edgeMBR

Figure 4: Sobel edge extraction from moving objects.


Max(pixel-by-pixel)

FO edgeMBR

BI edge

MI edgeMBR

Figure 5: Max-operations and MI edgeMBR.

where m(x, y) and s(x, y) are the mean and standarddeviation of the mask centered at the pixel (x, y), respectively,R is the maximum value of the standard deviation (in graylevel image, R = 128), k is the parameter with positive valuesin the range [0.2, 0.5], and Thsuppress is a suppression termand its value is set to be 50 empirically in this paper.

We then applied tfinal(x, y) to the following binarizationstep at location (x, y) according to (3) once tfinal(x, y) hadbeen obtained.

BinIMBR(x, y

) =⎧⎨

⎩

0 if St edgeMBR(x, y

) ≤ tfinal(x, y

),

255, otherwise,(3)

where BinIMBR represented the result after binarization. Theresult of using the binarization method on St edgeMBR wouldbe given in Figure 9. Another advantage of using adaptivebinarization methods instead of taking a fixed threshold wasthat users would not have to manually set a proper thresholdfor each video scene. And that should be a significant factorfor automatic monitoring systems.

We also had to remove the outer borders of BinIMBR

because of the following problems. In Figure 9, the shadowregion and real foreground object with the same motionvectors would make them always adjacent to each other. Also,the interior region of shadowed/foreground objects shouldbe homogeneous/nonhomogeneous (nontexture or edge-less/dominant edged), which implied that the edges fromshadows would appear at the outer borders of foregroundobjects. Considering these two properties, the objective ofremoving shadows could be treated as eliminating the outerborders and preserving the remaining edges which belongs toreal foreground objects. Also, the latter property mentionedabove might not be always satisfied. The interior regionof shadows would be sometimes little textured (e.g., lanemarkings) like the example shown in Figure 10. We couldsolve this kind of problem by the procedures that we havementioned earlier. From Figure 11(b), although the interiorregion of shadows had few edge-points after binarization, wecould easily cope with these noise-like points only by using aspecific filter after our subsequent processing procedures.

Figure 6: The result of subtracting BI edge from MI edgeMBR.

Figure 7: The result of subtracting BI edge from MI edgeMBR.

As mentioned above, we used a 7 × 7 mask toachieve boundary elimination. According to what have beenobserved, the widths of edge detected in shadows are in factvery approaching regardless of the edges that are extractedfrom far or near perspectives in the foreground images.And the width of edge is almost less than the length of 3pixels in most conditions. As Figure 12 illustrated, we put thegreen mask on the binarized edge points (marked as yellowcolor) of BinIMBR, and then scanned every point in BinIMBR.If the region covered by the mask completely belongs toforeground objects (marked as white point), we reserve thispoint (marked as red color); otherwise, we eliminate thispoint (marked as light blue point). After applying the outerboundary elimination, we could obtain the features for non-shadow pixels, notated as Ft EdgebasedMBR. In Figure 13, weshowed an actual example with Ft EdgebasedMBR expressedas red points.

3.2. Foreground-Pixel Extraction by Gray Level-Based ShadowRemoval. We tried to integrate the foreground-pixel extrac-tion by gray level-based approach into our shadow removalalgorithm, and this novel arrangement could enhance andstabilize the performance of that only by edge-based scheme.Figure 14 showed the flowchart of gray level-based shadowremoval foreground pixel extraction. Worthily speaking, wedeveloped a modified “constant ratio” rule by Gaussianmodel. We selected some pixels which belong to shadow-potential regions from foreground objects, calculated thedarkening factors as our training data, and then built aGaussian model for each gray level. Once the Gaussianmodel was trained, we could use this model to determine if


(a) Foreground Object (b) FO edgeMBR

(c) Background Image (d) BI edge

(e) St edgeMBR

(f) Subtract BI edge from FO edgeMBR

Figure 8: Examples of the ground surface with textures.

(a) Binarization of Figure 6 (b) Binarization of Figure 8(e)

Figure 9: Examples of BinIMBR.

each of the pixels inside foreground objects belonged to theshadowed region or not.

Here we would simply introduce the original “constantratio” rule so as to illustrate our modifications. Some studies[12, 18–20], using the property of “constant ration” for

shadow detection, expressed the intensity of each pixel on thecoordinate (x, y) in terms of I(x, y) with (4):

I(x, y

) =∫

e(λ, x, y

)ρ(λ, x, y

)σ(λ)dλ, (4)


(a) Foreground object (b) BinIMBR

Figure 10: Homogeneous property of shadows, example 1.

(a) Moving object (b) BinIMBR

Figure 11: Homogeneous property of shadow, example 2.

where λ is the wavelength parameter, e(λ, x, y) is theillumination function, ρ(λ, x, y) is the spectral reflectance,and σ(λ) is the sensitivity of camera sensor. The terme(λ, x, y) indicated the difference between nonshadowed andshadowed regions. For backgrounds, the term e(λ, x, y) iscomposed of the direct and diffused-reflected light com-ponents, but in the shadowed area, e(λ, x, y) only containsthe diffused-reflected light components. This differenceimplies the constant ratio property. Equation (5) showsthe ratio of I sh(x, y) and Ibg(x, y) where I sh(x, y) andIbg(x, y) represent the intensities of a shadowed pixel anda nonshadowed background pixel, respectively, and α iscalled the darkening factor which will be a constant over thewhole image.

I sh(x, y

)

Ibg(x, y

) = α. (5)

3.2.1. Gaussian Darkening Factor Model Updating. AsFigure 15 showed, we would rather stimulate the darkeningfactor with respect to each gray level by one Gaussian Modelthan that with respect to each pixel. In the beginning, wewould select the shadow-potential pixels as the updatingdata of Gaussian models by our three predefined conditionsintroduced in the followings.

(1) Pixels must belong to the foreground objects, forthe shadowed pixels must be part of the foregroundsideally.

(2) The intensity of a pixel (x, y) in the current frameshould be smaller than that in the frame of back-grounds, for the shadowed pixels must be darker thanbackground-pixels.

(3) The pixels obtained from the foreground-pixelextraction by edge-based shadow removal should beexcluded to reduce the number of pixels which mightbe classified as nonshadowed pixels.

For the practical case shown in Figure 16, the red pixels werethe selected points for Gaussian model updating.

After the pixels for updating were selected, we wouldupdate the mean and standard deviation of Gaussian model.Figure 17 displayed the flowchart of the updating process ofGaussian darkening factor model. The darkening factor αkcould be calculated as in (6):

I selected(x, y

)

Ibg(x, y

) = αk, (6)

where I selected(x, y) is the intensity of the selected pixel at(x, y), and Ibg(x, y) is the intensity of the background-pixelat (x, y). After calculating the darkening factor, we wouldupdate the kth Gaussian model.

We set a threshold as a minimum number of updatingtimes, and the updating times of each Gaussian modelmust exceed this threshold to ensure the stability of each


Mask

BinIMBR

(a) Using mask to scan BinIMBR

Foreground image

(b) Mask has covered the nonforeground region

Foreground image

(c) Mask is inside the foreground region

Ft EdgebasedMBR

(d) Final result of outer border elimination

Figure 12: Illustrations of boundary elimination.

model. Besides, in order to reduce the computation loadingof updating procedure, we gave a limit that each Gaussianmodel could only be updated at most 200 times for oneframe.

3.2.2. Determination of Non-Shadowed Pixels. Here, weintroduced how to extract the nonshadowed pixels by using

Boundaryelimination

BinIMBR

Foreground image

Ft EdgebasedMBR

Figure 13: An example of boundary elimination.

the trained Gaussian darkening factor model. Figure 18 gaveus the rules to calculate the difference between the mean ofGaussian model and the darkening factor, and check if thedifference was smaller than 3 times of standard deviation. Ifyes, the pixel would be classified as shadowed. Otherwise, itwould be considered as the nonshadowed pixel and could bereserved as a feature point.

Figure 19 described our tasks to determine the nonshad-owed pixels. If the kth Gaussian model was not trained,we would go checking if the nearby Gaussian models weremarked as trained or not. In our programs, we selectedthe nearby 6 Gaussian models for checking, and we chosethe nearest one if there existed any trained Gaussianmodel. Figure 20 gave an example where the pixels labeledby red color were the extracted feature pixels after ourdetermination task, and we denoted the set of these pixelsas Ft DarkeningFactorMBR.

4. Feature Combination

We combined two kinds of features which have beenintroduced in the Section 3 in our algorithm to extract fore-ground objects in a more accurate way. Figure 21 exhibitedthe flowchart of our feature combination. We integratedthese two features by applying “OR” operations. Figure 22demonstrated a real example of processed images by ourcombined features, called the feature-integration images inthis paper.

After acquiring the feature-integration images, we wouldlocate the real foreground objects, namely, the foregroundobjects excluding the shadowed regions. Hence, we usedthe connected-component-labeling approach with the min-imum bounding rectangle to orientate the real foregroundobjects. What is different from the common applications, wewould make some necessary preprocessing procedures beforeapplying the connected-component-labeling procedure. Thepreprocesses consisted of filtering and dilation operations.Both the median filter operation and morphological dilationoperation was conducted just once. In Figure 23, we couldsee that there existed some odd points in the left part


Gray level-based shadowremoval foreground

pixel extraction

Shadow-potentialregion extraction

Gaussian darkeningfactor model

updating


foreground pixels

Non-shadowpixel extraction

Foregroundobject

Gray level-basedshadow removal

foreground pixels

Figure 14: Flowchart of gray level-based shadow removal foreground pixel extraction.

Gray level:0 1 2 · · · 254 255

0 1Darkening

factor

0 1Darkening

factor

0 1Darkening

factor

Figure 15: An illustrated figure, each gray level has a Gaussianmodel.

if {WidthMBR k < Thmin MBR Width} AND{HeightMBR k < Thmin MBR Height}

Eliminate kth Minimum Bounding Rectangle;end

Algorithm 1

of the feature-integration images. It was obviously dueto the influences by land markings (see Figure 23(a)). InFigure 23(b), these pixels could be eliminated after thefiltering procedure. We then applied dilation operations tothe results after filtering in order to concentrate the leftfeature pixels, as shown in Figure 24.

After that, the connected-component-labeling approachcould be applied on the dilated images. As for our definedrules for this procedure, we used the minimum boundingrectangle for each independent region. Then, if any twominimum bounding rectangles are close to each other, wewill merge these two rectangles. Finally, iteratively check andmerge the previous results till no rectangle can be mergedwith.

After labeling and grouping, we would use the size filterto eliminate the minimum bounding rectangle of which thewidth and height were both smaller than a threshold, asdepicted in Algorithm 1. The subscript “k” indicated thekth minimum bounding rectangle. Figure 25 showed someexamples of the final located real objects; the green and lightblue rectangles revealed the foreground objects and finallocated real objects, respectively.

5. Experimental Results

In this section, we would demonstrate the results of ourproposed shadow removal algorithm. We implemented ouralgorithm on the platform of PC with P4 3.0 GHz and 1 GBRAM. The software we used is Borland C++ Builder onWindows XP OS. All of the testing inputs are uncompressedAVI video files. The resolution of each video frame is 320 ×240. The average processing time is 13.84 milliseconds foreach frame.

5.1. Experimental Results of Our Shadow Removal Algorithm.In the followings, we showed our experimental results underno occlusion situation in different scenes. In comparisonwith the results without using our algorithm, we used “red”and “green” rectangles to indicate the detected objects afterprocessing with and without applying our shadow removalalgorithm, respectively. In Figure 26, we could see that theproposed algorithm indeed successfully detected the realobjects and neutralized the negative influences by shadows,since the intensity of shadowed regions was low enoughto be distinguished from that of backgrounds. Besides, ourproposed algorithm could also cope with the larger-scaleshadows and provide the satisfactory results for differentsizes of vehicles like trucks which would be shown inFigures 26(d) and 26(f). In Figure 27, we demonstrated theprocessed results for different kinds of shadows such as


(a) Foreground object (b) Red points are the selected pixels

Figure 16: The selected pixels for Gaussian model updating.

A selectedpixel at (x, y)

Darkening factorcalculation

Ibgk (x, y)-th

Gaussian modelupdating

Gaussiandarkening

factor model

Figure 17: Gaussian darkening factor model updating procedure.

mi

3∗ STDEVi

Darkening factor

0 1

N(mi, σ2i )

Figure 18: Illustration of determination.

the smaller-scale shadows (shown in Figure 27(a)), the lowcontrast between intensities of shadows and backgrounds(shown in Figure 27(c)), and both the smaller-scale andlowly contrastive shadows (shown in Figure 27(e)). Clearly,our proposed method could work robustly no matter howlarge and how observable the shadows would be. Also inFigure 28, we gave the testing results of another scene wheremotorcycles and riders were precisely detected (shown inFigures 28(c) and 28(d)). Figure 29 provided the comparedresults processed by our proposed algorithm with those bythe representative methods which could have been accessible

through internet for the highway sequences of videos, whichshowed a much better processed result by our proposedalgorithm (right columns).

5.1.1. Occlusions Caused by Shadows. Here, we woulddemonstrate some examples of occlusions caused by theinfluences of shadows. In Figures 30(a), 30(e), 30(g) and30(i), two vehicles (or motorcycle—vehicle) were detectedin the same rectangle due to the influences of shadows, andFigure 30(c) indicated an even worse case in which vehicleswere framed together on account of light shadows. From allthe figures in the right column of Figure 30, it was apparentto see, our method could correctly detect the foregroundobjects under the influences of shadows. Moreover, we couldstill have the correct results (shown in Figure 30(j)) in theforeground object detection for a more difficult case (shownin Figure 30(i)) that three shadowed regions were detectedas one foreground object. That was to say, our proposedalgorithm could handle the problems of occlusions causedby shadows which have been always considered as toughtasks.

5.1.2. Discussions of Gray Level-BasedMethod. In Section 3.2,we used darkening factors to enhance the performance andreliability of the proposed algorithm. We hence in heremade the comparisons of experimental results by apply-ing and not applying our proposed approach mentionedin Section 3.2. Figure 31 showed a conspicuous examplein which green/red rectangles represented the foregroundobjects/detected objects. Figures 31(a), 31(c), and 31(e) werethe results without using the foreground-pixel-extractionapproach by gray level-based shadow removal. In otherwords, the only feature for extracting foreground objects was


Foreground object

Scan a pixelfrom foreground

image

Darkening factorcalculation

Ibgk (x, y)-th

Gaussian modelis stable

No

Yes

Find a nearbystable Gaussian

model

No

Yes

Shadowdetermination

No

YesPreservethis pixel

NoEnd scan

Yes

Gray level-basednon-shadow

foreground pixels

Figure 19: Flowchart of the nonshadowed pixel determination.

(a) Foreground Object (b) Ft DarkeningFactorMBR

Figure 20: An example of gray level-based nonshadow foreground-pixel extraction.



foreground pixels

Gray level-basedshadow removal

foreground pixels

Feature combination

Integration

Noise filteringand dilation

Labelingand grouping

Size filter Object list

Figure 21: Flowchart of feature combination.

(a) Foreground Object (b) Ft EdgebasedMBR

(c) Ft DarkeningFactorMBR (d) Feature integration image

Figure 22: An example after integration.

(a) Feature integration image (b) After filtering

Figure 23: Filtering by median filter for feature-integration images.


Figure 24: The result of dilation operations on Figure 23(b).

Table 1: Descriptions of testing videos for vehicle counting.

Testing video Scene Shadow description Video FPS

Video1 Highway Obvious and Large 30

Video2 Highway Light and Large 30

Video3 Expressway Obvious and Large 25

obtained from the foreground-pixel-extraction approachby edge-based shadow removal that we introduced inSection 3.1. But it would bring about the failure of detectionswhen the objects were edgeless (or textureless). As theimages shown in the left column of Figure 31, the car roofscould be regarded as edgeless, which might result in thatthe detected object were in pieces or that only the rearbumper could be detected. Figures 31(b), 31(d), and 31(f)exhibited the better results which were obtained from ourintroduced structure of feature combination including theinformation of edge-based and gray level-based shadowremoval.

5.2. Vehicle Counting. To prove the validity and versatilityof the proposed approach, we tried to apply our developedalgorithm to vehicle counting, one of the popular applica-tions in ITS. We had 3 testing videos for vehicle counting,and the scenes and properties of shadows in each video werearranged in Table 1. Figure 32 showed the scene of each videofor vehicle counting. In order to illustrate the compared andstatistical results in a more convenient way, we had Video1in 6 sectors, Video2 in 13 sectors, and Video3 in 2 sectors,respectively. Each of the sectors was about 2 minutes. We had4 lanes for both Video1 and Video2, and 2 lanes for Video3.In Table 2, we gave the number of passing vehicles on eachlane for each video manually.

We calculated the accuracy rate for each sector by (7) togive the comparisons in a more reasonable manner.

Accuracy rate =⎡

⎣1−⎛

⎝

∣∣∣Nprogram −Nmanual

∣∣∣

Nmanual

⎞

⎠

⎤

⎦× 100%,

(7)

where Nmanual and Nprogram represented the number of vehi-cles which were obtained from manual operations and ourprograms, respectively. As Table 3 showed, we could obtain

the compared consequences of average accuracy rates byusing the foreground object-detection results with/withoutapplying our proposed algorithm as the inputs in the exper-iments of vehicle counting. Table 4 indicated the averageaccuracy rate for each video in all lanes. From the testingresults, we would also list two kinds of failed examples inFigure 33 to indicate some erroneous detected results inthe vehicle counting application, which may be reasonableto illustrate the failed conditions in a more quantizedmanner. One possible condition which might result in thefalse consequences came from the much more complicatedtextures within the detected shadows and was illustratedin Figure 33(a). Figure 33(a) revealed that the algorithmfailed to provide a correct result in vehicle counting due tothe overcomplicated edge information reflected in detectedshadows. The major reason can be easily observed that theshadow of some specific object was not successfully detectedand eliminated. In fact, the research of shadow detection andremoval based upon all the processes by image processingonly has been a tough issue since we did not try to makeuse of any other information from some useful sensorsbut cameras. Moreover, this paper aimed at developinga practical algorithm in image processing procedures toefficiently remove the shadowing effect before dealing withthe applications of ITS, which would have less impact onthe performance of shadow removal and make the influencesdependent on some specific application. Owing to the rareappearance of such conditions in a longer recorded file ofvideos, the detected error rate in counting vehicles could bekept lower to a satisfactory range. As for the other possiblecondition that might result in the false detection results,Figure 33(b) illustrated this kind of example and also showedthe false detection consequence in the vehicle countingapplication. This occlusion case for two cars caused the falsecounting result because the shadows would be too unap-parent to be correctly detected and the two detected objectswere moving simultaneously. This kind of problem shouldbe categorized to another research field of image processing,yet not the issue that we have focused on in this paper. Sincethis phenomenon may result from many other conditionssuch as the dynamic behavior of moving vehicles, discussionsof different shadows, and influences of light reflection, wewould rather concentrate on developing a more practicaland automatic algorithm in shadow removal. Those exper-imental results revealed that our proposed algorithm couldnot only work well in vehicle counting but also improvethe performance of any applications under constraints ofshadows.

6. Conclusions

We in this paper present a real-time and efficient movingshadow removal algorithm based on versatile uses of GMM,including the background removal and development offeatures by Gaussian models. Our algorithm innovatesto use the homogeneous property inside the shadowedregions, and hierarchically detects the foreground objectsby extracting the edge-based, gray level-based features, and


(a) (b)

Figure 25: Examples of final located real objects.

(a) (b)

(c) (d)

(e) (f)

Figure 26: Experimental results of foreground object detection.

Table 2: Number of passing vehicles in each lane.

Testing video Partition numberLane1 Lane2 Lane3 Lane4

(vehicles) (vehicles) (vehicles) (vehicles)

Video1 6 102 189 116 89

Video2 13 464 505 373 261

Video3 2 58 75 — —


(a) (b)

(c) (d)

(e) (f)

Figure 27: Experimental results for different kinds of shadows.

Table 3: Vehicle counting results.

Testingvideos

Compared methodsLane1 Lane2 Lane3 Lane4

(Average Accuracy rate) (Average Accuracy rate) (Average Accuracy rate) (Average Accuracy rate)

Video1Without ShadowRemoval

81.58% 97.50% 96.57% 82.29%

With ProposedAlgorithm

100% 99.02% 97.22% 100%


92.88% 96.27% 95.55% 89.51%


97.59% 99.31% 99.68% 99.26%


95.16% 97.14% — —


100% 100% — —


(a) (b)

(c) (d)


(a) (b)

(c) (d)

(e) (f)



(a) (b)

(c) (d)

(e) (f)

(g) (h)

(i) (j)

Figure 30: Experimental results under the occlusion situation.


(a) (b)

(c) (d)

(e) (f)

Figure 31: Compared results of not applying and applying gray level-based shadow removal foreground-pixel extraction.

(a) Video1 (b) Video2 (c) Video3

Figure 32: Scenes of videos for vehicle counting.

(a) (b)

Figure 33: Some failed examples of image frames for (a) the much-texture case and (b) the special occlusion case.


Table 4: Average accuracy of all lanes in each video.

Testing Video Average Accuracy Rate

Video1Without Shadow Removal 89.49%

With Proposed Algorithm 99.06%


With Proposed Algorithm 98.96%


With Proposed Algorithm 100%

feature combination. Our approach can be characterizedby some original procedures such as “pixel-by-pixel maxi-mization”, subtraction of edges from background images inthe corresponding regions, adaptive binarization, boundaryelimination, the automatic selection mechanism for shadow-potential regions, and the Gaussian darkening factor modelfor each gray level.

Among all these proposed procedures, “pixel-by-pixelmaximization” and subtraction of edges from backgroundimages in the corresponding regions deal with the problemswhich result from the shadowed regions with edges. Adaptivebinarization and boundary elimination are developed toextract the foreground-pixels of nonshadowed regions. Mostsignificantly, we propose the Gaussian darkening factormodel for each gray level to extract nonshadow pixelsfrom foreground objects by using the information of graylevels, and integrate all the useful features to locate the realobjects without shadows. Finally, in comparison with theprevious approaches, the experimental results show that ourproposed algorithm can accurately detect and locate theforeground objects in different scenes and various types ofshadows. What’s more, we apply the presented algorithm tovehicle counting to prove its capability and effectiveness. Ouralgorithm indeed improves the results of vehicle countingand it is also verified to be efficient with the promptprocessing speed.

Acknowledgment

This work was supported in part by the Aiming for theTop University Plan of National Chiao Tung University, theMinistry of Education, Taiwan, under Contract 99W962, andsupported in part by the National Science Council, Taiwan,under Contracts NSC 99-3114-E-009 -167 and NSC 98-2221-E-009 -167.

References

[1] W. Zhang, X. Z. Fang, X. K. Yang, and Q. M. J. Wu, “Movingcast shadows detection using ratio edge,” IEEE Transactions onMultimedia, vol. 9, no. 6, pp. 1202–1214, 2007.

[2] R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, “Detectingmoving objects, ghosts, and shadows in video streams,” IEEETransactions on Pattern Analysis and Machine Intelligence, vol.25, no. 10, pp. 1337–1342, 2003.

[3] M.-T. Yang, K.-H. Lo, C.-C. Chiang, and W.-K. Tai, “Movingcast shadow detection by exploiting multiple cues,” IET ImageProcessing, vol. 2, no. 2, pp. 95–104, 2008.

[4] A. Cavallaro, E. Salvador, and T. Ebrahimi, “Shadow-awareobject-based video processing,” IEE Proceedings: Vision, Imageand Signal Processing, vol. 152, no. 4, pp. 398–406, 2005.

[5] K.-T. Song and J.-C. Tai, “Image-based traffic monitoring withshadow suppression,” Proceedings of the IEEE, vol. 95, no. 2, pp.413–426, 2007.

[6] N. Martel-Brisson and A. Zaccarin, “Learning and removingcast shadows through a multidistribution approach,” IEEETransactions on Pattern Analysis and Machine Intelligence, vol.29, no. 7, pp. 1133–1146, 2007.

[7] A. J. Joshi and N. P. Papanikolopoulos, “Learning to detectmoving shadows in dynamic environments,” IEEE Transac-tions on Pattern Analysis and Machine Intelligence, vol. 30, no.11, pp. 2055–2063, 2008.

[8] A. J. Joshi and N. Papanikolopoulos, “Learning of movingcast shadows for dynamic environments,” in Proceedings ofthe IEEE International Conference on Robotics and Automation(ICRA ’08), pp. 987–992, May 2008.

[9] A. Leone, C. Distante, and F. Buccolieri, “A texture-basedapproach for shadow detection,” in Proceedings of the IEEEConference on Advanced Video and Signal Based Surveillance(AVSS ’05), pp. 371–376, September 2005.

[10] M. Mohammed Ibrahim and R. Anupama, “Scene adaptiveshadow detection algorithm,” Proceedings of World Academyof Science, Engineering and Technology, vol. 2, pp. 1307–6884,2005.

[11] J.-W. Hsieh, S.-H. Yu, Y.-S. Chen, and W.-F. Hu, “Automatictraffic surveillance system for vehicle tracking and classifica-tion,” IEEE Transactions on Intelligent Transportation Systems,vol. 7, no. 2, pp. 179–187, 2006.

[12] C. Benedek and T. Sziranyi, “Bayesian foreground and shadowdetection in uncertain frame rate surveillance videos,” IEEETransactions on Image Processing, vol. 17, no. 4, pp. 608–621,2008.

[13] M. Xiao, C.-Z. Han, and L. Zhang, “Moving shadow detectionand removal for traffic sequences,” International Journal ofAutomation and Computing, vol. 4, no. 1, pp. 38–46, 2007.

[14] C. Stauffer and W. E. L. Grimson, “Adaptive backgroundmixture models for real-time tracking,” in Proceedings of theIEEE Computer Society Conference on Computer Vision andPattern Recognition (CVPR ’99), vol. 2, pp. 246–252, June 1999.

[15] P. Suo and Y. Wang, “An improved adaptive backgroundmodeling algorithm based on Gaussian mixture model,” inProceedings of the 9th International Conference on SignalProcessing (ICSP ’08), pp. 1436–1439, October 2008.

[16] J. Sauvola and M. Pietikainen, “Adaptive document imagebinarization,” Pattern Recognition, vol. 33, no. 2, pp. 225–236,2000.

[17] F. Shafait, D. Keysers, and T. M. Breuel, “Efficient implemen-tation of local adaptive thresholding techniques using integralimages,” in Document Recognition and Retrieval XV, vol. 6815of Proceedings of SPIE, San Jose, Calif, USA, January 2008.

[18] P. L. Rosin and T. Ellis, “Image difference threshold strategiesand shadow detection,” in Proceedings of the 6th BritishMachine Vision Conference, 1994.

[19] Y. Wang, K.-F. Loe, and J.-K. Wu, “A dynamic conditional ran-dom field model for foreground and shadow segmentation,”IEEE Transactions on Pattern Analysis andMachine Intelligence,vol. 28, no. 2, pp. 279–289, 2006.

[20] I. Mikic, P. C. Cosman, G. T. Kogut, and M. M. Trivedi,“Moving shadow and object detection in traffic scenes,” inProceedings of the 15th International Conference on PatternRecognition, vol. 1, pp. 321–324, 2000.

Documents

An Efficient and Robust Moving Shadow Removal Algorithm and Its