OBJECT TRACKING IN CLUTTER AND PARTIAL OCCLUSION THROUGHRULE-DRIVEN UTILIZATION OF SNAKES

Gabriel Tsechpenakis, Kostas Rapatzikos, Nicolas Tsapatsoulis and Stefanos Kollias

Image, Video and Multimedia Systems Laboratory,School of Electrical and Computer Engineering,

National Technical University of Athens,9 Iroon Polytechniou Str., 15773 Athens, Greece

ABSTRACT

Efficient moving object tracking in cases of partial occlu-sion is a challenging task for the researchers in the fieldsof computer vision and video processing. In modern codingstandards, like MPEG-4 and MPEG-7, the term of video ob-jects is used to define moving objects in a video sequence.Automatic extraction of such objects is by no means triv-ial, and occlusion is one of most important problems. Thispaper reformulates one of the most popular deformable tem-plates for shape modeling and object tracking, the Snakes,in a probabilistic manner, in order to include providencefor partial occlusion of the moving objects. Experimentsof object tracking in partial occlusion, in complex naturalsequences, where temporal clutter, abrupt motion and ex-ternal lighting changes have been carried out, showing theefficiency of the proposed approach.

1. INTRODUCTION

The deformable templates known as active contours havecome up and drawn special attention in the last decade, dueto their efficiency in dealing with problems like image andmotion segmentation [7], object detection, localization andtracking in video sequences [8, 10]. A major category of ac-tive contours, the Snakes, has been successfully applied toa variety of problems, such as edge detection and tracking[3]. Snakes are based on energy minimization along a curve;that is a curve deforms its shape so as to minimize an “in-ternal” and an “external” energy along its boundary. How-ever, Snakes are not problem free models: parameter tuning,shape initialization and the complexity of the energy mini-mization procedure are problems related to the applicabilityof Snakes, while the relatively poor performance in imagesand video sequences with complex and noisy background ismainly due to the fact that Snakes are one of the most rep-resentative examples of boundary-based active contours.Object tracking in partial occlusion is a problem which seemsto have similarities with object tracking in complex back-

ground, but it is actually much harder. Partial occlusionhides some parts of the object and in this case the external(image dependent) energy term, which contributes to the to-tal snake energy, has to be ignored. The basic issue here is tounderstand when partial occlusion occurs. In our approachthe movement history of the snake is used a predictor of itsposition in the next frame. Prediction serves two purposes:(a) it provides a reasonable shape initialization and (b) it canbe used for detecting partial occlusion.The probabilistic approach of tracking contours that we use,utilizes a snake model, which is based on both boundaryand region information. Region motion information aboutthe past and the present motion of the tracked object is pro-vided by a motion estimation scheme [1] and, based on theobject’s motion history, a shape prior knowledge (boundary-based), indicated asuncertainty region, is extracted. Themoving contour is estimated inside that region, according toan energy function. A force-based implementation, follow-ing a steepest descent-alike method is used to approximatethe global minimum solution of the snake energy function.Furthermore, the obtained solution should obey several con-straints in order to identify the boundaries of occluded partsof the moving objects.

2. THEORETICAL BACKGROUND

Active contour models were first introduced by Kass et al.[5] through the so-called snake models. In general, snakesconcern model and image data analysis, through the defini-tion of a linear energy function and a set of regularizationparameters. This energy function consists of two parts, theexternal energy termEext, which depends on the image dataaccording to a chosen criterion and the internal energy termEint, which enforces smoothness along the snake. The goalis to minimize the total snake energy and this is achieved it-eratively, after considering an initial estimate for the objectshape.The proposed total energyEsnake of the snakeCsnake, is

given by Esnake = Eint + Eext, whereEint is definedin terms of snake local curvatureCUsnake and elasticityDVsnake, whereasEext is defined with the use of a modifiedimage gradientGm, replacing the commonly used laplacian-of-gaussian term|∇Gσ∗I|, which introduces noise in thesnake models. More information about the definitions ofthe proposed energy terms can be found in [9].Before applying the tracking model in the current frame ofa sequence, as described in the following section, we pre-process the image to eliminate noise, with the use of a mor-phological Alternating Sequential Filter (ASF), based onsuccessive morphological area opening and closing opera-tions with structure elements of increasing scale and differ-ent orientations [6]. The main advantage of such filters isthat they preserve line-type image structures (edges), whichis impossible to be achieved, for example, with median fil-tering. The modified image gradientGm used for our pur-poses is actually a part of the Watershed transformation inimage segmentation problems [6] and consists of the ex-traction of binary image markers, through a morphologi-cal geodesic erosion reconstruction of the image gradient,and successive morphological conditional erosions of thesemarkers, so that they constitute the only local minima of theimage gradient.

2.1. Motion Estimates Extraction

The correct extraction of moving edges in terms of positionand direction is important and aids the accurate estimationof an object’s position from the current to the next frame.Several existing techniques are able to adequately cope withthe difficult problem of optical flow recovery given that theirassumptions hold. The challenge is to achieve high robust-ness against strong assumption violations commonly met inreal sequences. We adopt the motion estimation techniqueproposed by Black et al. [1] as an efficient tool for over-coming these violations. They reformulate the objectivefunction, which consists of the optical flow equation and thespatial coherence constraint, in order to include the robuststatistics tools [4] in an almost straightforward way. Theysimply take the standard least-squares formulation of opti-cal flow and use a robust estimator instead of the quadraticone. This approximation is then minimized using a coarse-to-fine (multiresolution) simultaneous over-relaxation tech-nique. The proposed reformulation results in an area-basedregression technique that is robust to multiple motions dueto occlusion, transparency or specular reflections and com-pensates for over-smoothing and noise sensitivity.

3. OBJECT TRACKING

The main issues that the proposed tracking approach is calledupon to cope with, are: (a) non-rigid (deformable) moving

objects, (b)moving objects with a complicated (not smooth)contour, (c) object movements which are not simple trans-lations, but also include rotations and motion towards thecamera capturing the sequence, (d) sequences with strongexistence of temporal clutter and external lighting changes,(e) sequences with complex background (common case innatural sequences), and (f) moving object partial occlusions.The issues (a)-(d) are covered by the formulation of proba-bilistic approach of snakes as it is shown in [9]. The issues(e) and (f) are the subjects of the current work and are dis-cussed in the following sections.

3.1. Force-Based Approach

Given the proposed snake model presented in Section 2, thefirst step is to extract some regions around the curve, whichare described asuncertainty regions. This is achieved byexploiting the motion history of the tracked contour (curvepoints’ motion in previous time instances), estimated withthe use of the motion estimation scheme proposed in sub-section 2.1: the previously estimated contour is deformedaccording to the previously estimated point motion and thestandard deviation of each point’s mean motion is calcu-lated; the uncertainty region around each point is then de-fined in terms of its corresponding standard deviation. Thenext step is to find the new position of each point of thecurve, inside its corresponding uncertainty region, whichcorresponds to the minimum of a criterion, which is definedby the snake’s energy terms described Section 2:

C(I+1) = arg minr∈R

[w1 ·D(r)CU +w2 ·D(r)

DV +w3 ·E(r)ext], (1)

whereD(r)CU andD

(r)DV are the differences between the cur-

vatures and elasticities of the contourC(I) and the curver ∈ R at thek-th point, respectively. The set of all possiblecurvesR emerge by oscillating the points of the curve in-side the uncertainty regionU. Finally, the weightsw1 andw2 are computed according to the curvatureCUsnake andelasticityDVsnake distributions’ zero crossings, that denotehow smooth and elastic the curve is, whilew3 is computedin terms of the mean value ofGm inside the uncertainty re-gionU, that denote how smooth (homogeneous) this regionis.The minimization of the equation (1) is a procedure of highcomplexity. This is a common problem when applying theminimization procedure of snakes and therefore we adopt aforce-based approach, as described in the following, insteadof using a dynamic programming algorithm. According tothat approach, energy terms are converted in forces resultingto the minimizing of them. The initial estimation of the ob-ject’s contourC(I+1)

init = [C(I+1)init,k |k = 1 . . . N ] in the frame

I + 1 is computed based on contour’s current location and

its instant motion, i.e.,

C(I+1)init = C(I) + m(I)

c , (2)

wherem(I)c = [m(I)

(c,1), ...,m(I)(c,N)] is the estimated motion

of the contour points in the frameI (obtained using the tech-nique described in 2.1). Then, the internal snake forces aredefined as,Fc(k) = w1(k) · [CU(C(I))(k)−CU

(C(I+1)init )

(k)] ·

n(I+1)k andFd(k) = w2(k)·[DV(C(I))(k)−DV

(C(I+1)init )

(k)]·

t(I+1)k , wheret(I+1)

k andn(I+1)k are the tangential unit and

normal vectors of the curveC(I+1)init at each pointk, respec-

tively. The forcesFd(k) represent the stretching compo-nents (elasticity), whereasFc(k) are the deformations ofthe curve along its normal directions (local curvature).Given the snake’s uncertainty regionU, for each pointkwe define a functiongm,k(p), constituting of all pixelspof Gm, inside the uncertainty regionU, lying on the linesegment that is defined by the normal direction of the curveC(I+1)

init at pointk:

gm,k(p) = [Gm(p)|(C(I+1)init,k−p)n(I+1)

k = 1, p ∈ U] (3)

pk = arg maxp

[gm,k(p)] , (4)

pk determines the most salient edge-pixel in the line seg-ment defined above and thus defines the direction of the ex-ternal snake force

Fe(k) = w3(k) · sgnk · eext(k) · n(I+1)k , (5)

wherew3(k) = ‖k − pk‖, k is the pixel corresponding tothe contour pointk, andsgnk denotes the direction of theforce: sgn = ‘ + ‘ if p is inside the curve, andsgn =‘ − ‘ otherwise. From the definition of the external en-ergy term it can be seen that it takes values close to zeroin regions of high image gradient (G2

m(k) ' 1) and valuesclose to unity in regions with relatively constant intensity(smooth)(G2

m(k) ' 0).In the force-based approach, the initialization of the exam-ined curveC(I+I)

init marches towards the object’s boundariesin the next frameI + 1, according to the forces applied toit. Thus, the minimization of equation (1) can be approx-imated by using the internal and external snake forces de-fined above, in an iterative manner similar to the steepestdescent approach [2]. The final contourC(I+1) is obtainedwhen one of the following criteria is satisfied: (a) if the re-sultant force is greater then the one of the next iteration, or(b) the maximum number of iterations is reached. It mustbe noted that the use of the proposed steepest descent ap-proach does not ensure that the final contour corresponds tothe solution of the equation (1), but under the constraints wepose, even ifC(I+1) corresponds to a local minimum, it isclose to the desired solution (global minimum).

3.2. Rule-driven Handling of Partial Occlusion

In order to separate background and object regions, espe-cially when the background in not homogeneous (smooth),as well as to cope with moving object’s partial occlusionthat may occur, we introduce more constraints thatpk (eq.(4)) must obey, so that its estimation will be reasonable.Without loss of generality, we suppose that the backgroundis static and possible occluding objects are also static. Letm

(I)c = 1

N

∑Ni=1 m

(I)c,i be the mean estimated motion of

the contour points at frameI andk− 1 andk + 1 be thesurrounding pixels of pointk on the line segment, alongwhich the functiongm,k is calculated, then (a)pk must di-vide that line segment in two parts: an immiscibly movingand a immiscibly static one, that isu(k− 1) ' m

(I)c and

u(k + 1) ' 0, or u(k− 1) ' 0 andu(k + 1) ' m(I)c , and

(b) pk must be a moving point with velocity close tom(I)c ,

whereu(·) denotes the instant velocity.Thus, taking the above constraints into consideration, weovercome cases such as (a) when the maximum is found inbackground: it is not a moving one and does not separatetwo immiscible (according to the motion) parts of the func-tion gm, (b) when the maximum is found inside the movingobject region: although it is a moving one, it does not dividethe functiongm in such two parts, (c) when occlusion oc-curs and the maximum is on the occluding object boundary:the maximum is not moving, although it makes the regiongm separation and (d) when occlusion occurs and the max-imum is in the occluding object region: neither the maxi-mum is moving, nor it makes such a separation. In thesecases, where these two constraints are not reached, we ig-nore the external force and evolve the curve according to itsinternal forces; in this way, we can obtain contours similarto the ones in the past frames.

4. EXPERIMENTAL RESULTS

In Figure 1, the proposed approach is applied to a stronglycluttered sequence, where the desired object is deformingalong the time; the accuracy of the method is based on thesnake’s external energy definition through the image mod-ified gradient utilization and the ASF pre-filtering. On theother hand, Figure 2 represents an indoor human head track-ing: despite the complicated background, the rule-basedconstraints proposed in subsection 3.2 separate correctly themoving head from background. Finally, Figure 3 illustratesa case of two moving objects, where, as time goes by, theone is getting partially occluded by a static obstacle, whilethe other is moving in the front of the obstacle. In sub-figures 3 (a1-d1), the motion estimates are illustrated, show-ing that the noise is effectively eliminated on the boundariesbetween the static and moving regions even when the oc-clusion occurs, whereas the respective sub-figures 3 (a2-d2)

(a) (b)

(c) (d)

Fig. 1. Example of a man walking in a cluttered sequence.

(a) (b)

(c) (d)

Fig. 2. Head tracking in with complicated background.

show that both objects’ contours are estimated with suffi-cient accuracy, due to the additional constraints in whichthe maximum of equation (4) is imposed.

5. CONCLUSIONS

In this work we have presented a rule-driven application ofSnakes for object tracking in partial occlusion and complexbackgrounds. Rules represent constraints on the continuityof the contour motion in successive frames as well as differ-ences in the mean motion of the object and the backgroundor the occluding object. Obtained results are encouraging,while the authors are working so as to include mathematicalrepresentation of the rules into the snake’s energy minimiza-tion function.

6. REFERENCES

[1] M.J. Black and P. Anandan, “The Robust Estima-tion of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields,”CVIU, 63(1), pp. 75-104, 1996.

[2] S. Haykin, “Neural Networks” Macmillan CollegePublishing Company,Section 5.3: The Steepest De-scent Method, pp. 124-126, 1994.

[3] M. Hoch and P. Litwinowicz, “A Practical Solution for

(a1) (a2)

(b1) (b2)

(c1) (c2)

(d1) (d2)

Fig. 3. Occlusion case: motion vectors and obtained objectcontours.

Tracking Edges in Image Sequences with Snakes,”TheVisual Computer, 12(2), pp. 75-83, 1996.

[4] P. Huber, Robust Statistics, Wiley eds., NY, 1981.

[5] M. Kass, A. Witkin and D. Terzopoulos, “Snakes: Ac-tive Contour Models,”Int. Journal of Comp. Vis., 1(4),pp. 321-331, 1988.

[6] V.K Madisetti and D.B Williams Eds.,The Digital Sig-nal Processing Handbook, CRC Press, 1998, Chap.74, pp. 20-26.

[7] A. Mansouri, T. Chomaud and J. Konrad, “A compar-ative evaluation of algorithms for fast computation oflevel set PDEs with applications to motion segmenta-tion,” In Proc. of ICIP’01, pp. 636 -639, vol.3, Thes-saloniki, Greece, 2001.

[8] N. Paragios and R. Deriche, “Geodesic Active Con-tours and Level Sets for the Detection and Trackingof Moving Objects,”IEEE Trans. on PAMI22(3), pp.266-280, 2000.

[9] G. Tsechpenakis, N. Tsapatsoulis and S. Kol-lias, “Probabilistic Boundary-Based Contour Trackingwith Snakes in Natural Cluttered Video Sequences,”submitted to Int. Journal of Image and Graphics(IJIG). http://www.image.ece.ntua.gr/∼gtsech/

[10] C. Vieren, F. Cabestaing and J.G. Postaire, “Catch-ing Moving Objects with Snakes for Motion Tracking”Pattern Recognition Letters, 16, pp. 679-685, 1995.