Stochastic approximation for background modelling

Computer Vision and Image Understanding 115 (2011) 735–749

Contents lists available at ScienceDirect

Computer Vision and Image Understanding

journal homepage: www.elsevier .com/ locate/cviu

Stochastic approximation for background modelling

Ezequiel López-Rubio ⇑, Rafael Marcos Luque-BaenaDepartment of Computer Languages and Computer Science, University of Málaga, Bulevar Louis Pasteur 35, 29071 Málaga, Spain

a r t i c l e i n f o a b s t r a c t

Article history:Received 8 March 2010Accepted 31 January 2011Available online 15 March 2011

Keywords:Background modellingProbabilistic mixture modelsStochastic approximationUnsupervised learning

1077-3142/$ - see front matter � 2011 Elsevier Inc. Adoi:10.1016/j.cviu.2011.01.007

⇑ Corresponding author.E-mail addresses: [email protected] (E. López-

(R.M. Luque-Baena).

Many background modelling approaches are based on mixtures of multivariate Gaussians with diagonalcovariance matrices. This often yields good results, but complex backgrounds are not adequately cap-tured, and post-processing techniques are needed. Here we propose the use of mixtures of uniform dis-tributions and multivariate Gaussians with full covariance matrices. These mixtures are able to cope withboth dynamic backgrounds and complex patterns of foreground objects. A learning algorithm is derivedfrom the stochastic approximation framework, which has a very reduced computational complexity.Hence, it is suited for real time applications. Experimental results show that our approach outperformsthe classic procedure in several benchmark videos.

� 2011 Elsevier Inc. All rights reserved.

1. Introduction

Research in intelligent video surveillance systems is an area inwhich scientific community has made great efforts for the lastyears, conceiving new ideas, developing new theoretical frame-works and implementing new algorithms which have stimulatedthe increase of the number of quality contributions [1–3].

This kind of systems can be applied to a wide range of potentialapplications, such as security and safety systems in importantbuildings, traffic surveillance in cities and highways, or supervisionof suspicious behaviour in supermarkets or in public means oftransport. In fact, any area which requires applications with realtime monitoring of motion objects from a camera video stream issusceptible of using intelligent video surveillance systems. How-ever, the increase of the research in video surveillance has notled to a huge rise of intelligent surveillance systems in operation.Although several commercial frameworks have already been de-fined to deal with motion objects both in outdoor and indoorscenes, nowadays, passive supervision systems keep being usedin many areas, in which the number of cameras exceeds the capa-bility of human operators to analyse and monitor them.

Several steps are required for a typical video surveillance sys-tem to reach its objective. At first, a video object segmentationmethod is required to obtain the objects in motion of the stream.Subsequently, a tracking algorithm is applied to identify the ob-jects in several frames of the sequence. A matching between eachblob (or set of blobs) and each tracked object must be done. Finally,an algorithm to detect the object behaviour is used to understand

ll rights reserved.

Rubio), [email protected]

and analyse what it is happening in a scene. In general, the firststage is considered as the crucial part of the system since the restof modules depend largely on it. In addition to that, low time com-plexity is demanded at object detection stage in order to carry outthe entire process in real time.

Every video surveillance system starts its activity by detectingmoving objects in the scene. A comparison among different tech-niques can be found in some important works [4–6]. In essence,the aim of this task is to separate pixels corresponding to the fore-ground motion objects from those corresponding to the stationarybackground ones. Different kinds of approaches can be found in theliterature to model this problem, such as techniques based on opti-cal flow [7–9], whose main disadvantage is that it requires veryhigh computational time; frame difference, which is efficient butinaccurate and unreliable; or background subtraction, which mod-els the background by comparison with the frames of the se-quence. In particular, the development of a background modelhas been widely used by a large number of papers as a way to de-tect the foreground objects in each frame. This model can be ascomplex as it is required, from the simplest ones which are justbased on an image background, statistical parametric modelswhich represent the colour signal of each pixel, to even more ambi-tious models which analyse and maintain information about a setof scene features. The complexity of these background models di-rectly influence how complex or efficient their update process is.

However, it is not easy to find a low-complexity method whichis able to handle all the unexpected situations, gets suitable andunbeatable results and whose efficiency and robustness is high.The simplest process, which involves the subtraction of the currentanalysed frame from the previously calculated background image,is not enough to deal with a great number of difficulties whichmake the object detection process rather complex. Unfavourable

http://dx.doi.org/10.1016/j.cviu.2011.01.007

mailto:[email protected]

mailto:[email protected]


http://www.sciencedirect.com/science/journal/10773142

http://www.elsevier.com/locate/cviu

736 E. López-Rubio, R.M. Luque-Baena / Computer Vision and Image Understanding 115 (2011) 735–749

factors such as both abrupt and progressive illumination changesin the scene, cast foreground object shadows on the background,noisy frames because of a poor quality image source, or repetitivemovements of stationary objects like waving tree branches mustbe taken into account by the developed object detection method.

Different approaches in the literature try to find solutions to theprevious drawbacks. In [10], a combination between backgroundsubtraction and frame difference is presented, dealing with suddenillumination changes and noise reduction. The background updateis performed using a moving W-window and the foreground maskis obtained by using an ad-hoc thresholding process. Both param-eters, W and the threshold rS are empirically chosen depending onthe analysed scene. Another method which uses an sliding windowwith some recent past L frames is included in [11], where an esti-mation of the background image of the sequence is obtained bymeans of a temporal median background process over the window.A novel variation of frame difference is presented in [12], where amethod based on double frame differences and edge encloses is de-scribed, which stands out mainly in its reduced time complexity.There are two different low-level segmentation modules depend-ing on the lighting conditions, since the system is designed to oper-ate both daytime and at night. It is considered as a competitivealgorithm for detecting simple rigid objects only, like in trafficsequences.

A model based on the Kalman filter is developed in [13,14], withthe aim of compensating the illumination variability of each pixel.They replace the sliding window with an online adaptive computa-tion of the background model, whose update is controlled by alearning rate a. In [15], Wren et al. describes its Pfinder methodby modelling the background for each pixel using a normal distri-bution. As a modified version of this method, Stauffer et al. in[16,17], which complete and improve the probabilistic approachoutlined in [18], present a well-known theoretical framework forthe upgrade background, based on a mixing process of Gaussiandistributions for each pixel, using an expectation-maximisation(EM) algorithm to update them. This statistical approach is morerobust in scenes with many moving objects and lighting changes,and it is one of the most cited techniques in the literature. Becauseof the fact that sometimes it is not necessary to use a fixed numberof distributions, in [19] the number of Gaussians is estimated on-line, thus reducing slightly the time complexity of the algorithm.

In order to determine as fast as possible whether a pixel value tbelongs to the background distribution in the methods which mod-el pixels as probability density functions, a quick implementationof the probability p(tjBack) is performed for all the statisticalapproaches [15,16,19]. It is assumed that a background match isdefined as a pixel value within N standard deviations of a distribu-tion. This assumption is stated in [16,20]. This parameter N, whichis considered as a threshold for Gaussian tiles in standard devia-tions, varies depending on the amount of noise which is observedin the scene.

Unlike the previous parametric methods, Elgammal et al. [21]model the background using non-parametric methods, more ro-bust and invariant especially in outdoor scenes with a huge back-ground variability. Within the statistical methods, Haritaoglu et al.[22] build a model named W4, to represent each pixel with threevalues: minimum and maximum values, and the maximum inten-sity difference between consecutive frames during the trainingperiod. These three values are updated periodically. McKennaet al. [23] use an adaptive background model with colour and gra-dient information to reduce the influences of shadows and unreli-able colour cues. Under a Bayes decision framework, Li et al. [24]developed an approach which separates the analysis of the station-ary pixels from the moving objects in scenes with complex back-grounds, by using different feature vectors for each purpose. Areference background image is also maintained, by performing a

IIR filter in order to handle the gradual changes for stationary back-ground objects.

Other kind of approaches which analyse the input data by usinga phase-divided structure have been proposed in the literature[25]. These models combine different methods to solve some ofthe main problems associated to the segmentation task. Basically,they have a low-level layer which processes the data pixel by pixel,and then, several additional modules are applied in order to im-prove the segmentation quality. Morphological operations suchas closing and opening as well as connected components algo-rithms and the elimination of small objects should be used for thispurpose.

Our proposal departs from the classical algorithms based on theEM framework. We propose to use stochastic approximation meth-ods [26–35] which are based on the Robbins–Monro algorithm [36].These techniques have already been used to replace or modify typi-cal EM strategies in a wide range of problems [37–41], and they havebeen applied in image and video processing [42–44,29,45]. A keyadvantage of stochastic approximation is that it is conceived to pro-cess input samples online. On the other hand, EM is commonly asso-ciated with batch processing of a complete set of input data. In videosegmentation, the online processing nature of stochastic approxi-mation can be of paramount importance, since data is generated inreal time. Moreover, this framework has an intrinsic tendency toassign more relevance to the newer data with respect to the olderones, which reinforces its suitability for this task. As we will see, itis also a robust strategy which needs a minimal number of tuneableparameters.

The structure of the paper is as follows. Section 2 presents thenew video segmentation model, its learning rule and the initializa-tion and implementation details. Section 3 is devoted to experi-ments. First we explain our choice of the competing models.Then an evaluation methodology is developed, with both qualita-tive and quantitative performance measures. And, of course, theexperimental results are also presented. In Section 4 we discussthe novel features and the advantages of our method. Finally, con-clusions are drawn in Section 5.

2. The model

2.1. Model definition

We propose to use the stochastic approximation framework totrain a mixture which models the distribution of pixel valuest(x) = (t1(x), t2(x), t3(x)) at position x = (x1,x2). There is oneGaussian component for the background and one uniform compo-nent for the foreground. This leads to the following probabilisticmodel for the distribution of pixel values at any given position,where we drop the position indices x for the sake of simplicity:

pðtÞ ¼ pBackpðtjBackÞ þ pForepðtjForeÞ¼ pBackGðtjlBack;CBack þWÞ þ pForeUðtÞ ð1Þ

We have:

Gðtjl;RÞ ¼ ð2pÞ�D=2 detðRÞ�1=2 expððt� lÞTR�1ðt� lÞÞ ð2Þ

UðtÞ ¼1=VolðHÞ iff t 2 H

0 iff t R H

�ð3Þ

lBack ¼ E½tjBack� ð4Þ

lFore ¼ E½tjFore� ð5Þ

CBack ¼ E½ðt� lBackÞðt� lBackÞT jBack� ð6Þ

E. López-Rubio, R.M. Luque-Baena / Computer Vision and Image Understanding 115 (2011) 735–749 737

8i 2 fBack; Foreg;Rni ¼ PðijtnÞ ¼pipðtnjiÞ

pBackpðtnjBackÞ þ pForepðtnjForeÞð7Þ

where H is the support of the uniform pdf, Vol(H) is the D-dimen-sional volume of H, and W is a constant diagonal matrix which ac-counts for the quantization noise due to lossy compression (seeSection 2.3). In background modelling, we typically have D = 3 fortristimulus pixels, and H spans all the colour range:

H ¼ fðt1; t2; t3Þjt1; t2; t3 2 ½0;v �g ð8Þ

with Vol(H) = v3. In most cases the pixels are given in RGB, but tcould also be expressed in any other colour space, such as L⁄u⁄v⁄.On the other hand, if we have colour values with 8 bit precisionthen v = 255. It must be highlighted that lFore is not used to com-pute the probability densities in any way. It is only needed whena sudden background change is detected, as explained in Section2.4.

The Gaussian component G(tjl,R) is able to capture a widerange of dynamic backgrounds because CBack (and hence CBack + W)are not restricted to be diagonal matrices. On the other hand, theuniform component U(t) is able to model foreground objects ofany colour equally well.

2.2. Learning

When applied to the probabilistic model given in the previoussubsection, the Robbins–Monro stochastic approximation algo-rithm yields the following update equations at time step n for anobserved pixel value tn (see Appendix A for details):

8i 2 fBack; Foreg;piðnÞ ¼ ð1� �Þpiðn� 1Þ þ �Rni ð9Þ8i 2 fBack; Foreg;miðnÞ ¼ ð1� �Þmiðn� 1Þ þ �Rnitn ð10Þ

8i 2 fBack; Foreg;liðnÞ ¼miðnÞpiðnÞ

ð11Þ

MBackðnÞ ¼ ð1� �ÞMBackðn� 1Þ þ �Rn;Backðtn � lBackðnÞÞðtn � lBackðnÞÞ

T ð12Þ

CBackðnÞ ¼MBackðnÞpBackðnÞ

ð13Þ

where � is a constant step size, � � 0.01. The step size is constant(and not decaying) because the system is time varying, as consid-ered in Section 3.2 of [26]. Please note that mi and MBack are auxil-iary variables required to update the model parameters pi, li andCBack.

2.3. Quantization noise estimation

The diagonal noise matrix W models the effects of the quantiza-tion and the compression on the pixels. This is estimated sepa-rately because these effects appear and disappear suddenly fromone frame to the next, so that they cannot be captured adequatelyby the stochastic approximation learning algorithm. Furthermore,we assume that these effects are approximately equal for all thepixels, so that an offline estimation of a global and constant valueof W is carried out. The matrix is given by

W ¼r2

R 0 00 r2

G 00 0 r2

B

0B@

1CA ð14Þ

The noise parameters may be estimated by comparing the original,uncompressed pixels s(x) with the observed (compressed) pixelst(x) for a selection of videos:

8i 2 fR;G;Bgr2i ¼ E½ðsiðxÞ � tiðxÞÞ2� ð15Þ

where the expectation is computed by averaging over all the pixelsin the video selection. Please note that this estimation is carried outoffline (that is, W is a constant throughout the online learning). Inpractice, it is often found that the original values s(x) cannot be ob-tained because the video is available in compressed format only.This problem is solved by estimating the original values by first-or-der kernel regression of the observed values t(x). In this context theoriginal image is regarded as a function of the position x:

s : ½1;NumRows� � ½1;NumCols� ! ½0;v �3 ð16ÞsðxÞ ¼ ðs1ðxÞ; s2ðxÞ; s3ðxÞÞ ð17Þ

Please note that s(x) = s(x1,x2) is defined even for fractional valuesof x1 and x2. We remove the quantization noise by means of theNadaraya–Watson kernel smoother (first-order kernel regression):

bsðxÞ ¼P

y2WðxÞKxðyÞtðyÞPy2WðxÞKxðyÞ

ð18Þ

where bsðxÞ is the estimated original value at pixel position x, W(x)is a bidimensional window centered at position x, and Kx is asmoothing kernel function. The adaptation to the input data is en-hanced if we choose the kernel to depend on the local gradientcovariance matrix Sx [46,47]:

Kx yð Þ ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffidetðSxÞ

p2ph2 exp � 1

h2 ðx� yÞT Sxðx� yÞ� �

ð19Þ

Here h is a global smoothing parameter, and Sx is given by [48]:

Sx ¼ Ex½ðrtÞðrtÞT � ð20Þ

Ex½n� ¼Z Z

VðxÞnðyÞdy ð21Þ

wherert is a 2 � 3 Jacobian matrix and V(x) a local neighbourhoodof pixel x. In practice V(x) is a circle with fixed radius r centered in x,and Sx is quickly estimated by Sobel or Prewitt edge detection filtersapplied to the luminance component of t. That is, rt is reduced to2 � 1 size.

The final step of this procedure is the approximation of thequantization noise by comparison of the estimated original valuessi with the observed (compressed) values ti:

8i 2 fR;G;Bgr2i � E½ðsiðxÞ � tiðxÞÞ2� ð22Þ

where the expectation is computed by averaging over all the pixelsin the video selection, as before.

2.4. Sudden background change detection

Sometimes an object integrates into the background, for exam-ple when a car is parked. This produces a sudden change in thebackground distributions of many pixels, i.e. those which are insidethe integrating object. These changes must be dealt with sepa-rately [24]. As pointed out by Kushner and Yin in Section 3.2 of[26], stochastic approximation of time varying systems is onlyadvisable when the rate of change of the estimated parameters isslow. Hence, these sudden changes must be tackled with a differ-ent procedure which detects these situations that the stochasticapproximation framework is unable to cope with.

The detection procedure needs a previous offline study. Let v bethe ground truth for a particular pixel:

v ¼1 iff the pixel belongs to the foreground0 iff the pixel belongs to the background

�ð23Þ

Then, let ~v be the estimation of v carried out by the basic algo-rithm, i.e. without sudden background change detection:

1 Further information about the source code and test videos is available in http://www.lcc.uma.es/%7Eezeqlr/backsa/backsa.html. The test videos are also available aselectronic annexes to this paper.

2 http://staff.science.uva.nl/%7Ezivkovic/DOWNLOAD.html


~v ¼

1 iff the pixel is classified as foregroundby the basic algorithm

0 iff the pixel is classified as backgroundby the basic algorithm

8>>><>>>: ð24Þ

The basic algorithm works correctly if and only if ~v ¼ v. We de-fine three disjoint random events: undetected background change(Change), correct foreground detection (Good) and other situationsthat we are not interested in (Other):

Change � ð~v ¼ 1Þ ^ ðv ¼ 0Þ ð25ÞGood � ð~v ¼ 1Þ ^ ðv ¼ 1Þ ð26ÞOther � ~v ¼ 0 ð27Þ

At frame n, the number of frames z(n) that the pixel has beencontinuously classified as foreground is defined as follows:

zðnÞ ¼maxfNj~vðnÞ ¼ ~vðn� 1Þ ¼ . . . ¼ ~vðn� NÞ ¼ 1g ð28Þ

Please note that z(n) is readily computed by a running counterwhich is incremented each frame that ~v ¼ 1 and reset to zero eachtime that ~v ¼ 0. The offline task consists in estimating the proba-bilities PðGoodjz; ~v ¼ 1Þ and PðChangejz; ~v ¼ 1Þ. This is accom-plished by counting the number of times that Change and Goodare verified for a certain value of z, where we take as input samplesthe values of z(n) for all frames n and all pixels such that ~v ¼ 1.These pixel counts are noisy functions of z that are smoothed bykernel regression before estimating the probabilities.

Then we compute a threshold Z:

Z ¼min zjPðChangejz; ~v ¼ 1Þ > 12

� �ð29Þ

The computation of Z finishes the online study. When the onlinesystem is operating, in each pixel we maintain a running counter ofthe number of frames z. If we have at frame n that a pixel is clas-sified as foreground and z(n) P Z, then that pixel must be reset.The pixel reset procedure involves setting CBack = 0. Additionally,we swap lBack with lFore, and mBack with mFore.

2.5. Initialization

Here we outline the initialization procedure to be carried out atthe beginning of the execution of the method. The first K frames ofthe analysed video sequence are used for this purpose, withK = 200 in our experiments. The background model of every pixelis set up according to the K pixel value samples obtained fromthose first frames. Hence we have:

8i 2 fBack; Foreg;pið0Þ ¼12

ð30Þ

mBackð0Þ ¼pBackð0Þ

K

XK

j¼1

tj ð31Þ

lBackð0Þ ¼mBackð0ÞpBackð0Þ

ð32Þ

MBackð0Þ ¼pBackð0Þ

K

XK

j¼1

ðtj � lBackð0ÞÞðtj � lBackð0ÞÞT ð33Þ

CBackð0Þ ¼MBackð0ÞpBackð0Þ

ð34Þ

On the other hand, the initialization of lFore is not critical, sinceits value is not used for the first Z frames at least. Hence we startwith a point in the middle of the colour space:

lForeð0Þ ¼v=2v=2v=2

0B@

1CA ð35Þ

mForeð0Þ ¼ pForeð0ÞlForeð0Þ ð36Þ

Finally, the running counter is initialized with z(0) = 0.

2.6. Implementation

Since our system is designed to operate in real time, it is of par-amount importance to reduce the number of required calculationsas possible. In particular, the computation of the determinant andthe inverse of R, which are needed to obtain the responsibilities Rni

are the heaviest parts in terms of floating point operations. Nextwe outline how an optimised implementation should be built.

First of all, there is no point in using routines from standardnumerical libraries, since they are tuned for high dimension D. Bestresults are obtained by deriving specific equations for the D = 3case, which is the only one to be considered. In addition to this,we should take advantage of the fact that R is symmetric, becauseit is a covariance matrix.

The computation of the determinant is carried out by imposingthe symmetry constraint to the well-known Sarrus’ scheme:

deta b c

b d e

c e f

0B@

1CA ¼ adf � c2dþ 2bce� e2a� b2f ð37Þ

Once the determinant is known, we can compute the inversefrom the cofactors. Again, we take into account the symmetry ofR, which implies that R�1 is also symmetric:

a b c

b d e

c e f

0B@

1CA�1

¼ deta b c

b d e

c e f

0B@

1CA

0B@

1CA�1

a0 b0 c0

b0 d0 e0

c0 e0 f 0

0B@

1CA ð38Þ

a0 ¼ df � e2 ð39Þb0 ¼ ce� bf ð40Þc0 ¼ be� cd ð41Þd0 ¼ af � c2 ð42Þe0 ¼ bc � ae ð43Þf 0 ¼ ad� b2 ð44ÞThese equations provide a way to fulfil real time executionrequirements.

3. Experimental results

In this section a principled evaluation methodology is devel-oped in order to compare our stochastic approximation approach(AE)1 against several state-of-art video segmentation methods. Theresults of the segmentation were assessed both quantitatively andqualitatively using a dataset of diverse sequences with a wide rangeof complex backgrounds and different situations.

3.1. Methods

Considering that our approach is included among the so-calledpixel-level methods, which analyse the scene at a low level pixelby pixel, the comparison is done with respect to several techniquesof the same class. The Pfinder method (PF) proposed in [15], thetwo Mixture of Gaussians approaches, GMM and AGMM, whichare developed in [16] and [19] respectively, and the Li et al.approach [24] called FGD, are also included in our performanceanalysis. The AGMM code is given by the author in its web page2,

http://www.lcc.uma.es/%7Eezeqlr/backsa/backsa.html

http://www.lcc.uma.es/%7Eezeqlr/backsa/backsa.html

http://staff.science.uva.nl/%7Ezivkovic/DOWNLOAD.html

(a) Frame (b) GT (c) AE (d) FGD (e) AGMM (f) GMM (g) PF

Fig. 1. Experimental results on scenes with high variability on the background. An outdoor campus scene (CAM) with waving tree branches corresponds to the first three rowswith frames 1392, 1758, and 2348. A meeting room environment (MR) with wavering curtains because of air-conditioning is shown in the last rows with frames 2774 and3266. From left to right, each column shows: frame, GT, AE, FGD, AGMM, GMM and PF results.

(a) Frame (b) GT (c) AE (d) FGD (e) AGMM (f) GMM (g) WREN

Fig. 2. Experimental results on complex scenes with stationary moving objects. A subway station environment (SS) with moving escalators is associated to the first row(frame 4277). Two different scenes with moving water, water surface (WS) and fountain (FT), are shown in the last rows, in which the frames are 1196, 1499 and 2605,respectively.


whereas an FGD implementation version is easily accessible in thefree OpenCV Computer Vision library3.

3 http://sourceforge.net/projects/opencvlibrary.html

The implementations for all of the above considered methodsare based on C language MEX files for its use with Matlab, as isthe case of the implementation of our method. MEX files carryout the most time consuming sections, while the rest is imple-mented with Matlab scripts. We have not used any GPU resources,

http://sourceforge.net/projects/opencvlibrary.html


so there is no need for special graphics hardware. All the experi-ments reported on this paper have been carried out on a 32-bitPC with a quad core 2.40 GHz CPU and 3 GB RAM.

(a) Frame (b) GT (c) AE (d) FG

Fig. 3. Experimental results on level crossing environments (LC), which are in the first thrthe fourth row with frame 390, and synthetic sequences which model 3D objects over rea748 and 804, respectively.

(a) Frame (b) AE (c) FGD

Fig. 4. Experimental results in three long sequences: BusStop (BUS) showing the frame 2left to right, each column shows: frame, AE, FGD, AGMM, GMM and PF results.

It is important to notice that as our aim is to assess the outputsof the five compared methods as fairly as possible, no post-processing technique is taken into account, due to the fact that

D (e) AGMM (f) GMM (g) PF

ee rows with frames 324, 430 and 465, scenes from corridors in leisure centres (OC),l backgrounds (V2 and V4), corresponding to the last four lines, with frames 332, 611,

(d) AGMM (e) GMM (f) PF

780, Intersection (IS), frame 2090, and US Highway 101 (US) with frame 7125. From

Table 1Parameter settings tested during evaluation of the object detection algorithms. Thecombination of all the values of the parameters generate a set of configurations foreach method.

Algorithm Test parameters

PF [15] Learning rate, a = {1e�4, 5e�4, 0.001, 0.005, 0.01,0.05, 0.1}Threshold for Gaussian tiles in standard deviations,N ¼ f2;2:5;3;5g

GMM [16], AGMM [19] Learning rate, a = {1e�4, 5e�4, 0.001, 0.005, 0.01,0.05, 0.1}No. of Gaussian mixture components, K = {3,5,7}Threshold for Gaussian tiles in standard deviations,N ¼ f2;2:5;3;5gWeight threshold, T ¼ f0:6;0:8g


these modules are independent of the background model and nofeedback on the model is performed from their outputs.

3.2. Sequences

The comparison of the five previous algorithms is performedusing a representative set of nine outdoor and indoor video se-quences. Some valuable public repositories provide these datasetstogether with the ground truth information, which consists of a setof manually segmented images where the sets of pixels belongingto the foreground and the background are defined. One of themwas created by the International Conference VSSN’06, on the occa-sion of an algorithm competition in Foreground/Background Seg-mentation.4 These datasets are considered as synthetic datasetssince they are composed by motion objects generated by 3D soft-ware together with real background scenes. A notorious advantageof this kind of datasets is that the ground truth can be automati-cally defined for all the frames, because the foreground objectsare added subsequently to the real scene. The sequences namedVideo2 (V2) and Video4 (V4) have been selected for ourcomparison.

Several sequences from the dataset developed by Li et al. [24]and available in his website5 have been used in our study, becausethey are considered as complex scenes which contain some of themost common problems which the ideal object detection methodshould deal with. Thus, each one of the scenes has a wide varietyof characteristics and presents a significant challenge because ofits complexity. Sequences which contain moving objects as a partof the stationary background, sudden illumination changes, fore-ground object’s pixel features which may be subsumed by the mod-elled background, cast shadows or foreground motionless objectsappearing at the beginning of the process, are a significant sampleof the problematic situations which arise in video surveillance. Sev-eral of the available segmented sequences have been included,namely: meeting room with moving curtain (MR), campus withplentiful vegetation moving continuously (CAM), water surface(WS), public fountain throwing water (FT) and moving escalators ina subway station (SS).

Another representation of the analysed sequences in the litera-ture is obtained from CAVIAR dataset.6 Only a sequence where peo-ple are walking on the corridor (OS) is included in our test data dueto the difficulty with getting the ground truth. The assessment ofan outdoor level crossing scene (LC) is also carried out.

In order to check the stability of our approach, we have addedthree new video sequences which have several thousand frameseach. The BUS sequence shows a busy bus stop during the morningwhereas the IS sequence is an example of a highway intersection.7

Another traffic long scene (US) provided by NGSIM8 is used in ourevaluation. Unfortunately, the ground truth is not available for thesehuge sequences. Because of this, the experimental results on theselong scenes are only qualitative.

3.3. Qualitative results

The performance results of the five studied methods are ob-served in Figs. 1–3. Each row shows an comparison of the segmen-tation results for each method from a single frame. All the rowscorrespond to some frames from the selected datasets. The firstcolumn displays some frames from the different benchmark se-quences whereas in the second column the manually segmented

4 http://mmc36.informatik.uni-augsburg.de/VSSN06_OSAC.5 http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html.6 http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1/.7 http://limu.ait.kyushu-u.ac.jp/en/dataset/.8 http://ngsim-community.org/.

foreground mask (GroundTruth) is depicted. From the third col-umn to the seventh one, the segmentation results of the followingmethods are showed: our stochastic approximation method (AE), Liet al. approach (FGD), adaptive mixture of Gaussians (AGMM),standard mixture of Gaussians (GMM) and single Gaussian distri-bution (PF), respectively. Several consequences can be extractedfrom the images. In most cases, our approach (AE) achieves betterresults than the other alternatives, both indoor and outdoor scenes.Particularly, the amount of noise in the outdoor ones is drasticallyreduced in the AE and FGD methods in comparison to the rest ofstatistical approaches. The negative effect known as waving treesin this kind of sequences causes a continuous variation of the pixeltonality on the background, requiring models which can representdisjoint sets of these pixel values. This is rather significant in CAM(the first three rows in Fig. 1) where the noise is so intense that themotion objects are difficult to separate. A similar situation happensin LC frames (Fig. 3; first, second and third row) although is evenworse for the PF method, whose foreground results are unreliableand not suitable to find the motion objects exactly. Other examplesof the noise problem for AGMM, GMM and PF are observed in thesequences WS (the last two rows in Fig. 2) and V4 (the last tworows in Fig. 3).

On the other hand, repetitive movements of the stationarybackground objects keep appearing in the evaluated indoor scenes,either caused by escalators in the SS sequence (the first rowin Fig. 2) or by curtains in motion because of the air conditioningin a meeting room in the MR sequence (fourth and fifth rows inFig. 1). It can be seen that only our approach has correctly handledthis situation, by removing almost totally both the escalators andthe curtains from the foreground mask in the two sequences. Fur-thermore, the problem of camouflage, in which there is a significantsimilarity between the pixel colours from the background and fore-ground, is also presented in MR sequence (last row, Fig. 1). Theappearance of holes in the foreground mask is a clear example ofit, and its solution should be reached by analysing globally eachobject in the post-processing phase. With regard to the cast shad-ows, our approach together with the statistical ones (AGMM,GMM and PF) are more sensitive to them than the FGD method,incorporating these pixels into the foreground. This can be noticedin the OS sequence, where the people’s reflected image on the cor-ridor floor is considered as foreground (fourth row in Fig. 3). As weconsider this as a problem to be dealt with in a subsequent post-processing phase, we do not attempt to solve it in the experiments,in spite of the fact that it affects the assessment of our method neg-atively. In stationary scenes like V2 (fifth and sixth row in Fig. 3),

FGD [24] Learning rate, a1 = {0.01,0.05,0.1}a2 = {1e�4,5e�4,0.001,0.005,0.01, 0.05,0.1}a3 = {0.01,0.05,0.1}Similarity threshold, d = {2,2.5,3}

AE Step size, e0 = {1e�4,5e�4,0.001,0.005,0.01,0.05,0.1}

http://mmc36.informatik.uni-augsburg.de/VSSN06_OSAC

http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html

http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1/

http://limu.ait.kyushu-u.ac.jp/en/dataset/

http://ngsim-community.org/

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

WaterSurfacePFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FPFN

Video4PFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

Video2PFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

LevelCrossingPFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

OneShopOneWait1corPFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FPFN

FountainPFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

EscalatorPFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

CurtainPFGMMAGMMFGDAE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FP

FN

CampusPFGMMAGMMFGDAE

Fig. 5. False Positives (FP) and False Negatives (FN) ratios after applying each method to the test sequences with all the parameter configurations. Each coloured point ‘�’ isconsidered as a different configuration of that method. The closer the points are to the origin, the better the segmentation is. Additionally, the method is less sensible to aparameters’ change if the cloud of points keeps compact and grouped.

Table 2Quantitative evaluation using the accuracy measure (Eq. (45)) and comparison results over the test sequences. The results show the mean and standard deviation after analysingeach dataset, where the best performance for each sequence is highlighted in bold. Our approach is the best method in six of nine sequences.

PF GMM AGMM FGD AE

Campus (CAM) 0.22 ± 0.12 0.27 ± 0.13 0.26 ± 0.13 0.50 ± 0.22 0.60 ± 0.17Meeting room (MR) 0.66 ± 0.07 0.71 ± 0.08 0.71 ± 0.09 0.83 ± 0.05 0.78 ± 0.05Subway station (SS) 0.31 ± 0.10 0.41 ± 0.09 0.40 ± 0.09 0.28 ± 0.12 0.46 ± 0.12Fountain (FT) 0.56 ± 0.08 0.60 ± 0.08 0.59 ± 0.08 0.46 ± 0.16 0.43 ± 0.17Level crossing (LC) 0.53 ± 0.15 0.77 ± 0.06 0.75 ± 0.08 0.78 ± 0.05 0.88 ± 0.03Corridor (OC) 0.71 ± 0.03 0.70 ± 0.04 0.70 ± 0.05 0.75 ± 0.02 0.72 ± 0.04Video2 (V2) 0.86 ± 0.03 0.91 ± 0.02 0.91 ± 0.02 0.76 ± 0.06 0.92 ± 0.02Video4 (V4) 0.38 ± 0.07 0.49 ± 0.07 0.49 ± 0.07 0.66 ± 0.09 0.68 ± 0.11Water surface (WS) 0.77 ± 0.03 0.80 ± 0.03 0.80 ± 0.03 0.78 ± 0.06 0.87 ± 0.02



there is not much difference between the segmentation quality inall the assessed techniques as we can observe. However, a nondesirable effect is produced after applying the FGD method insequences with quick movements on the foreground, which leavesa trace of the motion objects in previous frames. This is particularlyevident in sequences MR, V2 and V4.

3.4. Performance measures

Many algorithms have been proposed for objective evaluationof motion detection methods. Since successful tracking reliesheavily on accurate object detection, the evaluation of objectdetection algorithms within a surveillance systems plays an impor-tant role in overall performance analysis of the whole system. Gen-erally, the selection of a set of evaluation metrics depends largelyon the kind of applied segmentation. Provided that the groundtruth is supplied, evaluation techniques based on comparison withthat ideal output can be further classified according to the type ofmetrics they propose. Typically, pixel-based metrics [49–51] con-sist of a combination of true positives (tp), false positives (fp), truenegatives (tn) and false negatives (fn), computed over each frameand whose sum corresponds to the image size. It should be notedthat fp and fn refer to pixels misclassified as foreground (fp) orbackground (fn), while tp and tn account for accurately classifiedpixels.

Other alternatives which use object-based metrics are found inthe literature [52]. They analyse features as false alarms, detectionfailures, or merge and split regions but always from the perspec-tive of an object as a whole, and not as a sum of a set of pixels.We consider this approach very interesting after applying post-processing techniques, in which the output is a set of clean blobs,far from noise and spurious objects. Additionally, these measurescan help to select the best post-processed configuration in orderto enhance the segmentation results [20]. However, and as we said

Table 3Another quantitative evaluation using the F-measure, given by a combination of the precstandard deviation after analysing each dataset, where the best performance for each seq

PF GMM

Campus (CAM) 0.34 ± 0.16 0.40 ± 0.17Meeting room (MR) 0.79 ± 0.05 0.83 ± 0.06Subway station (SS) 0.47 ± 0.12 0.58 ± 0.09Fountain (FT) 0.72 ± 0.07 0.74 ± 0.07Level crossing (LC) 0.69 ± 0.10 0.87 ± 0.04Corridor (OC) 0.83 ± 0.02 0.82 ± 0.03Video2 (V2) 0.93 ± 0.02 0.95 ± 0.01Video4 (V4) 0.55 ± 0.07 0.65 ± 0.06Water surface (WS) 0.87 ± 0.02 0.89 ± 0.02

Table 4Frames per second (FPS, higher is better) for each analysed sequence together with its framtime requirements are fulfilled if a method is over 15 fps. Each value is obtained by usinmethod and sequence. The best frame rate for each sequence is shown in italic.

Size AGMM FGD

BusStop (BUS) 320 � 240 99.10 ± 0.06 21.16 ±Campus (CAM) 160 � 128 338.80 ± 0.60 64.38 ±Meeting room (MR) 160 � 128 340.59 ± 0.87 84.94 ±Subway station (SS) 160 � 130 311.71 ± 0.26 60.33 ±Fountain (FT) 160 � 128 363.85 ± 1.40 89.83 ±Intersection (IS) 320 � 240 102.40 ± 0.06 24.11 ±Level crossing (LC) 360 � 240 74.51 ± 0.03 20.59 ±Corridor (OC) 384 � 288 67.76 ± 0.04 14.24 ±Video2 (V2) 384 � 240 84.93 ± 0.11 27.61 ±Video4 (V4) 384 � 240 76.31 ± 0.05 18.00 ±Water surface (WS) 160 � 128 376.46 ± 0.79 86.35 ±US Highway 101 (US) 640 � 480 23.86 ± 0.16 6.37 ±

before, the post-processing phase and its evaluation are out of thescope of this paper, so we use a subset of the pixel-based metrics asour performance measures.

In this paper a similarity standard measure based on the previ-ous pixel-based metrics is applied. It was defined as a metric tocheck the MPEG Segmentation Quality [53] and used as the start-ing point of other developed metrics [54]. It has also been usedto compare different object detection methods [24,25]. This quan-titative measure named spatial accuracy is defined as follows:

AC ¼ cardðA \ BÞcardðA [ BÞ ð45Þ

where ‘card’ stands for the number of elements of a set, A is the setof all pixels which belong to the foreground, and B is the set of allpixels which are classified as foreground by the analysed method:

A ¼ fxjvðxÞ ¼ 1g B ¼ fxj~vðxÞ ¼ 1g ð46Þ

In order to take into account some information about false neg-atives and false positives, the proportions of these metrics are alsodefined and normalised using the foreground mask:

FN ¼ cardðA \ BÞcardðA [ BÞ ; FP ¼ cardðA \ BÞ

cardðA [ BÞ ð47Þ

From set theory we know that:

AC; FN; FP 2 ½0;1� AC þ FN þ FP ¼ 1 ð48Þ

The optimal performance would be achieved for AC = 1, FN = 0,FP = 0. On the other hand, the worst possible performance corre-sponds to AC = 0, FN + FP = 1. Additionally, we utilise two informa-tion retrieval measurements, recall and precision, to quantify howwell each algorithm matches the ground truth, by taking into ac-count both the background and the foreground. Like the previousaccuracy measure, the new ones have also been rather appliedin vision for comparing the performance of motion detection

ision and recall values and defined in the Eq. (50). The results show the mean anduence is highlighted in bold.

AGMM FGD AE

0.40 ± 0.17 0.64 ± 0.23 0.74 ± 0.150.83 ± 0.06 0.90 ± 0.03 0.88 ± 0.030.56 ± 0.10 0.42 ± 0.15 0.62 ± 0.120.74 ± 0.07 0.62 ± 0.16 0.58 ± 0.170.85 ± 0.05 0.87 ± 0.03 0.94 ± 0.020.82 ± 0.03 0.86 ± 0.02 0.83 ± 0.030.95 ± 0.01 0.86 ± 0.04 0.96 ± 0.010.65 ± 0.06 0.79 ± 0.07 0.80 ± 0.080.89 ± 0.02 0.88 ± 0.04 0.93 ± 0.01

e size (first and second columns). The bigger the frame size, the fewer FPS ratio. Realg the parameter configuration with the best background detection accuracy for each

PF GMM AE

0.01 115.48 ± 0.13 32.40 ± 0.01 262.64 ± 18.230.51 413.81 ± 0.36 134.30 ± 0.11 1011.76 ± 9.520.09 431.94 ± 0.38 158.05 ± 0.12 1015.24 ± 13.100.06 415.96 ± 0.31 122.28 ± 0.11 992.86 ± 11.680.17 427.95 ± 0.58 154.38 ± 0.19 998.76 ± 18.800.02 116.19 ± 0.15 32.56 ± 0.02 270.90 ± 2.100.01 101.09 ± 0.32 34.67 ± 0.03 239.12 ± 3.730.00 80.25 ± 0.06 32.16 ± 0.04 164.33 ± 5.000.01 96.96 ± 0.20 45.84 ± 0.03 226.50 ± 1.230.00 94.38 ± 0.12 34.96 ± 0.03 226.89 ± 1.950.17 430.72 ± 0.61 188.06 ± 0.22 1003.81 ± 13.860.00 28.57 ± 0.04 9.94 ± 0.01 34.99 ± 0.06


algorithms. Both are considered as a standard within this scope.Therefore, the precision (PR) and recall (RC) measures can be com-puted as follows (higher is better):

PR ¼ cardðA \ BÞcardðBÞ RC ¼ cardðA \ BÞ

cardðAÞ PR;RC 2 ½0;1� ð49Þ

RC is defined as the percentage of the correctly detected pixelsto the real moving objects whereas PR is defined as the percentageof the detected pixels to all detected moving object pixels. High re-call means less segmentation misses while high precision meansless false segmentation. Often there is a trade-off between preci-sion and recall, i.e. we can make one of them grow as desired atthe expense of diminishing the other.

Finally, another standard measure that combines precision andrecall, the traditional F-measure or balanced F-score, is computed:

F-measure ¼ 2 � PR � RCPRþ RC

ð50Þ

Table 5Best configuration for each method over all the analysed sequences. The sequence namesmethods. The accuracy, precision and recall measures (third, fourth and fifth column) are lilast column. The row highlights in bold correspond to the best option for that sequence b

Sequence Method Accuracy Precis

Campus (CAM) PF 0.22 ± 0.12 0.27 ±GMM 0.27 ± 0.13 0.35 ±AGMM 0.26 ± 0.13 0.33 ±FGD 0.50 ± 0.22 0.58 ±AE 0.60 ± 0.17 0.67 ±

Meeting room (MR) PF 0.66 ± 0.07 0.76 ±GMM 0.71 ± 0.08 0.95 ±AGMM 0.71 ± 0.09 0.95 ±FGD 0.83 ± 0.05 0.94 ±AE 0.78 ± 0.05 0.95 ±

Subway station (SS) PF 0.31 ± 0.10 0.37 ±GMM 0.41 ± 0.09 0.52 ±AGMM 0.40 ± 0.09 0.49 ±FGD 0.28 ± 0.12 0.32 ±AE 0.46 ± 0.12 0.71 ±

Fountain (FT) PF 0.56 ± 0.08 0.82 ±GMM 0.60 ± 0.08 0.77 ±AGMM 0.59 ± 0.08 0.76 ±FGD 0.46 ± 0.16 0.54 ±AE 0.43 ± 0.17 0.61 ±

Level crossing (LC) PF 0.53 ± 0.15 0.67 ±GMM 0.77 ± 0.06 0.86 ±AGMM 0.75 ± 0.08 0.84 ±FGD 0.78 ± 0.05 0.92 ±AE 0.88 ± 0.03 0.91 ±

Corridor (OC) PF 0.71 ± 0.03 0.86 ±GMM 0.70 ± 0.04 0.79 ±AGMM 0.70 ± 0.05 0.78 ±FGD 0.75 ± 0.02 0.91 ±AE 0.72 ± 0.04 0.76 ±

Video2 (V2) PF 0.87 ± 0.03 0.93 ±GMM 0.91 ± 0.02 0.94 ±AGMM 0.91 ± 0.02 0.94 ±FGD 0.76 ± 0.06 0.84 ±AE 0.92 ± 0.02 0.94 ±

Video4 (V4) PF 0.38 ± 0.07 0.51 ±GMM 0.49 ± 0.07 0.63 ±AGMM 0.49 ± 0.07 0.61 ±FGD 0.66 ± 0.09 0.74 ±AE 0.68 ± 0.11 0.75 ±

Water surface (WS) PF 0.77 ± 0.03 0.93 ±GMM 0.80 ± 0.03 0.95 ±AGMM 0.80 ± 0.03 0.95 ±FGD 0.78 ± 0.06 0.95 ±AE 0.87 ± 0.02 0.95 ±

Overall evaluation metrics are found as their average over theentire video sequence.

3.5. Parameter selection

One of the major drawbacks of motion detection algorithms isto find the correct value of the defined parameters. Many timesit is rather complicated to tune them for a particular analysedscene. Although our model defines several parameters, we haveempirically found that a fixed value for some of them is suitablefor all the studied scenes. Specifically, the number of frames usedin the initialisation phase (Section 2.5) is always assigned toK = 200, and the global smoothing parameter (Section 2.3) is heldfixed to h = 2. Therefore, in our model only the step size parameter�0, which influences the learning process, must be tuned for eachscene.

Accordingly, for each parameter of the studied methods a validset of values is considered. Then multiple configurations of the

are shown in the first column whereas the second column one indicates the appliedsted for each method in the second column. The test parameter configuration is in theetween all the alternatives.

ion Recall Parameters

0.17 0.61 ± 0.12 N ¼ 5; a ¼ 0:10.19 0.62 ± 0.13 N ¼ 5; a ¼ 0:05; K ¼ 3; T ¼ 0:80.19 0.63 ± 0.12 N ¼ 5; a ¼ 0:05; K ¼ 3; T ¼ 0:80.27 0.83 ± 0.08 a1 = 0.05, a2 = 0.01, a3 = 0.01, d = 30.19 0.88 ± 0.06 e0 = 0.01

0.05 0.83 ± 0.07 N ¼ 3; a ¼ 0:00050.04 0.74 ± 0.08 N ¼ 5; a ¼ 0:0005; K ¼ 5; T ¼ 0:80.04 0.74 ± 0.08 N ¼ 5; a ¼ 0:0005; K ¼ 7; T ¼ 0:80.04 0.87 ± 0.05 a1 = 0.01, a2 = 0.0001, a3 = 0.05,d = 2.50.03 0.82 ± 0.06 e0 = 0.0005



0.22 0.76 ± 0.06 N ¼ 5; a ¼ 0:0050.08 0.88 ± 0.02 N ¼ 5; a ¼ 0:005; K ¼ 7; T ¼ 0:80.09 0.88 ± 0.02 N ¼ 5; a ¼ 0:005; K ¼ 7; T ¼ 0:80.05 0.84 ± 0.04 a1 = 0.01, a2 = 0.005, a3 = 0.05, d = 2.50.03 0.96 ± 0.01 e0 = 0.01

0.02 0.80 ± 0.03 N ¼ 5; a ¼ 0:0050.03 0.86 ± 0.04 N ¼ 5; a ¼ 0:005; K ¼ 3; T ¼ 0:60.03 0.87 ± 0.04 N ¼ 5; a ¼ 0:005; K ¼ 3; T ¼ 0:60.01 0.81 ± 0.03 a1 = 0.01, a2 = 0.001, a3 = 0.1, d = 2.50.02 0.92 ± 0.04 e0 = 0.01





methods are generated by combining the values of the parametersin every possible way. The considered parameter values for eachmethod are shown in Table 1. Fig. 5 shows the result of all the gen-erated configurations for each method in each one of the nine se-quences after applying the previously defined evaluation metrics.The horizontal axis corresponds to the average percentage of FPon the scene, while the vertical axis is associated with FN values.Each point of the plot represents the average FP and FN of a gener-ated configuration when applied to that sequence. The closer thepoints are to the origin, the better the segmentation has been car-ried out. The optimum performance occurs if FN = 0 and FP = 0,which implies there is a perfect match between the output of thealgorithm and the ground Truth (AC = 1), according to the relationbetween the three computed metrics. The results are always belowthe diagonal of the plot because we always have FN + FP 6 1, asseen in Eq. (48). Those configurations which are closer to thediagonal correspond to obtained segmentations with poor quality.As shown in the accuracy and F-measure (Tables 2 and 3, respec-tively), our approach (AE) gets the best results for six of the nineanalysed sequences (CAM, SS, LC, V2, V4 and WS), while for twoof the remaining ones our performance is quite competitive with

300 320 340 360 380 400 420 440 460 4800.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame Number

Prec

isio

n

LevelCrossing

PFGMMAGMMFGDAE

300 320 340 360 380

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Frame

Rec

all

LevelCr

300 350 400 450 500 550 600 650 700 750

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Frame Number

Prec

isio

n

Video2

PFGMMAGMMFGDAE

300 350 400 450 5000.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Frame

Rec

all

Vide

500 550 600 650 700 750 800 850

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame Number

Prec

isio

n

Video4

PFGMMAGMMFGDAE

500 550 600 650

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame

Rec

all

Vide

Fig. 6. Precision, recall and accuracy measures, which are shown frame by frame over thapproach.

respect to the other methods (MR and OS sequences). The resultsof the best performing configuration are shown for each sequencein Table 5, including three different quality measures and theparameter setting for each method.

Next we discuss the robustness of the methods with respect tothe choice of their parameters. Many times a wrong choice of theparameters of the applied segmentation algorithm generates adeficient segmentation, where it is not possible to distinguishmoving objects at first sight. Sometimes, it is not easy to determinein advance the appropriate parameters to get a good quality seg-mentation, which requires a tedious empirical assessment of thescene to assign these values. Fig. 5 shows the variability in theevaluation of each algorithm. This variability depends largely onthe analysed sequence, but the robustness of the method has alsoan influence, i.e. more robust methods yield smaller values. If theconfiguration cloud is compact, it means that the results do notvary significantly after a change in its parameters. On the otherhand, if several configurations are far from each other, it impliesthat the variation of a parameter causes abrupt changes in the re-sults, which is a very undesirable property for a motion detectionalgorithm. As shown in Fig. 5, the compactness for AGMM, GMM

400 420 440 460 480

Number

ossing

PFGMMAGMMFGDAE

300 320 340 360 380 400 420 440 460 4800.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame Number

Accu

racy

LevelCrossing

PFGMMAGMMFGDAE

550 600 650 700 750

Number

o2

PFGMMAGMMFGDAE

300 350 400 450 500 550 600 650 700 7500.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Frame Number

Accu

racy

Video2

PFGMMAGMMFGDAE

700 750 800 850

Number

o4

PFGMMAGMMFGDAE

500 550 600 650 700 750 800 8500.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame Number

Accu

racy

Video4

PFGMMAGMMFGDAE

e sequences LC, V2 and V4. It is remarkable the stability and low variability of our AE


and PF methods is rather poor in most of the sequences, while ourapproach and FGD model have their configurations very close to-gether. In other words, the performance of the proposed methodis not very sensitive to the parameter selection. For instance, in se-quences as LC and V2 our results are fairly clustered, while at thesame time they provide the best value of accuracy. The experimen-tal qualitative results on several long scenes (Fig. 4) further dem-onstrate the robustness of our proposal.

Table 6Comparative performance with respect to the colour space. The selected sequence is

3.6. Quantitative results

The previous figures have provided a qualitative assessment ofthe selected methods’ performance. In order to show more clearlythe improvements achieved by our AE approach, the correspondingquantitative evaluations are listed in Table 5, where a comparisonis provided for each sequence using a set of performance evalua-tion measures.

Tables 2 and 3 display a comparison among all the methods foreach sequence using the performance measures, AC and F-measure,respectively. As it can be noted, our method AE achieves the bestresults for six of the nine test sequences. We must say that theAC performance measure is faithful as long as there are foregroundobjects in the evaluated frame. Otherwise, the most probable con-figuration is FP = 1, FN = 0, AC = 0 because of the more than likelyoccurrence of false positives after the detection. Therefore, thisconfiguration does not truly reflect the effectiveness of the meth-ods since it is only based on the detected foreground and not inthe background. Moreover, in outdoor scenes with great back-ground variability, and where moving objects are quite smallcompared to the image size, the results of the assessment willsignificantly decrease due to this foreground importance. To han-dle this problem, only frames containing foreground objects areevaluated with the aim of not affecting the overall accuracy mea-sure adversely.

The accuracy, precision and recall results after the evaluationare represented in several figures and tables. In Fig. 6, three mea-sure graphs (Precision, Recall, and Accuracy, respectively) are plot-ted, in which the signal of each measure is shown along the timefor each analysed method. We must highlight the improvementof our approach (AE) in the first row of Fig. 6 (LC sequence), whereit gets higher values both in recall and accuracy measures. More-over, the three obtained signals for the method AE in the sequenceV2 (second row of the same figure) are much more stable thanthose corresponding to the other techniques like FGD. The resultsfor the sequence V4 are quite similar between AE and FGD methods,although the signals for the first one are less abrupt and slightlyhigher than for the last one. The quantitative results from thethree sequences represented in Fig. 6 are shown (formatted asmean ± std) in Table 5.

In Table 4 we show the results of the time complexity analysisfor each method. From the third column on, each cell shows theaverage rate of Frames Per Second (FPS) which each techniqueachieves when analysing each sequence (higher is better). It mustbe noted that the frame size influences the computational com-plexity of the algorithms. As said before, the processing time ismeasured on a 32-bit PC with a quad core 2.40 GHz CPU and3 GB RAM. As seen, our approach outperforms all its competitorsin every tested video sequence.

meeting room (MR).

Photo YCC RGB YCbCr YPbPr BestAcc �WorseAcc

PF 0.56 ± 0.13 0.66 ± 0.07 0.61 ± 0.11 0.64 ± 0.01 0.10GMM 0.57 ± 0.10 0.71 ± 0.08 0.61 ± 0.09 0.66 ± 0.09 0.14AGMM 0.57 ± 0.10 0.71 ± 0.09 0.61 ± 0.09 0.66 ± 0.09 0.14FGD 0.57 ± 0.11 0.83 ± 0.05 0.63 ± 0.09 0.63 ± 0.08 0.26AE 0.74 ± 0.07 0.78 ± 0.05 0.75 ± 0.07 0.83 ± 0.06 0.09

4. Discussion

The method we have just described departs in some fundamen-tal ways from those commonly found in literature. These differ-ences can be summarised as follows:

1. The fact that our method always uses full covariance matricesand Mahalanobis distances implies that it provides some equi-variance features with respect to affine transformations on theinput data [55–57] that proposals based on diagonal covariancematrices [16] do not have. This means that our method is moreindependent from the choice of the RGB, Y‘‘CbCr, Y’’PbPr, orPhoto YCC colour spaces, since all of them are obtained fromeach other by affine transformations [58]. That is, the back-ground detection results of our method are less dependent onthe used colour space, as we can observe in Table 6, in whichthe accuracy of each method after analysing the Meeting Roomsequence (MR) in different colour spaces is observed. In thisexample the difference between the better and worse accuracymeasures is found in the last column. The lower differencevalue, the more invariant the method to the colour space. Asseen, our method is more stable with respect to the used colourspace than all the others. It should be noted that the invarianceof a method with respect to the colour space is relevant only ifthe method is good enough in terms of efficiency. In the table,this implies that the stability of the PF method is useless, sinceit yields the worst background detection accuracy results. Nev-ertheless, the equivariance of our method cannot be complete,since the colour channels are encoded separately in the videofiles, so that the quantization errors do not transform in anequivariant way.

2. The stochastic approximation framework provides a formal jus-tification of the learning algorithm. This is not the case for somelearning rules which are heuristic modifications of the expecta-tion-maximisation (EM) method. These modifications areaimed to obtain an online scheme. However, it must beremarked that the EM method is originally designed for batchlearning. Hence, any online version of EM should have a clearformalisation, if the desirable properties of the original methodare to be preserved [59].

3. Our method models the foreground by means of a uniform dis-tribution. This is of paramount importance in order to cope withobjects which do not follow any colour pattern. Every objectwith a previously unseen colour is modelled by the foregroundmixture component just as well as any other object. Previousproposals rely on mixtures of Gaussian distributions for theforeground [60], so that new colours are poorly modelled. Inorder to illustrate this crucial point, Fig. 7 shows a comparisonbetween our model with one Gaussian and one Uniform mix-ture components, and the same model with two Guassian mix-ture components. This way, the advantages of the first versionare revealed, since the ‘Two Gaussians’ approach misses com-pletely the foreground pixels which are far from the peak,which leads to background detection failures. In addition tothis, Table 7 compares the accuracy values of the two algorithmversions for the sequences CAM and MR. It is clear from thetable that the results for the model with one Gaussian andone uniform distribution are quantitatively better than theother approach. Please note that the table also shows that ourquantization error estimator provides significant performanceimprovements.

Table 7Quantitative evaluation using the accuracy measure which shows the improvements of including the quantization error estimation (W) and using one Gaussian and one Uniformdistributions in the model.

Campus (CAM) Without W With W Curtain (MR) Without W With W

Gauss + Unif 0.57 ± 0.18 0.60 ± 0.17 Gauss + Unif 0.71 ± 0.14 0.78 ± 0.05Gauss + Gauss 0.48 ± 0.18 0.52 ± 0.17 Gauss + Gauss 0.69 ± 0.11 0.76 ± 0.06

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

X

p(X)

Histogram2Gauss PDF1Gauss + 1Unif PDF

0 10 20 30 40 50 60 70 80 90 100

64 65 66 67 68 69 70

0.002

0.004

0.006

0.008

0.01

0.012

0.014

Fig. 7. Analysis of the pixel (560, 205) over the sequence US Highway 101 (US). Two different versions of our approach are tested. The green line shows the modelrepresentation with one Gaussian and one Uniform mixture component, and the red line corresponds to modelling the scene using two Gaussian mixture components. Azoom view gives us a better idea of the difference. Please note that the first version (Gaussian + Uniform) is able to model the foreground pixel values in the tails of thehistogram much better than the version with two Gaussians. (For color interpretation in this figure legend the reader is refered to see the web version of this article.)


4. Since we only use two mixture components (a Gaussian and auniform), the computational complexity of the method isreduced with respect to those with multiple components [19](see the Table 4). Moreover, there is no need for an additionaldecision procedure [16] to determine which component shouldbecome part of the background when a sudden backgroundchange is detected. Consequently, the possible decision errorsin that procedure are avoided.

5. The comparison between different segmentation methods hasconcluded that our approach (AE) has been the best one interms of efficiency in six of the nine sequences, with the addi-tional advantage of being the second simplest applied method(after PF). Another outstanding feature is its robustness and sta-bility, with little variability in the segmentation results betweenconsecutive frames, so that it features a stable level of efficiencyover the time.

6. Unlike the rest of the comparative methods in which they haveto adjust between two (PF) and four parameters (GMM AGMMand FGD), only one should be adjusted in this model, whichinvolves simplifying the fine-tuning parameter setup andavoiding some errors because of a bad parameter adjustment.In fact, the initial choice of parameter values is a crucial issue

for the suitable performance of an object detection algorithm.This choice is clearly critical in AGMM, GMM and PF methods,where the modification of some parameters have caused signif-icant variations in the results. Although such variations alsodepends on the analysed datasets, our method minimises theimpact of a poor choice of the parameter values, getting similarresults by using different configurations of the algorithm, asseen in Fig. 5.

5. Conclusions

This paper proposes a novel statistical approach for backgroundmodelling and foreground object detection, in which one Gaussianand one uniform distributions are used to model the backgroundand foreground respectively. It features an update process of themodel that is based on stochastic approximation, which has beenproved to be suitable and effective as a learning method for realtime algorithms with discarding data.

Experiments have been carried on a variety of complex environ-ments, from outdoor and indoor scenes to synthetic sequenceswhich try to mimic real scenarios. These sequences have providedmanually segmented images (GT) in order to apply a set of


evaluation metrics which are used to compare the performance ofdifferent object detection algorithms.

As a result of this comparison between several of the most citedsegmentation techniques, it is noteworthy that our approach hasobtained the best results in six of the nine test sequences, whilehas been competitive in the rest of them. Additionally to that, anexhaustive study about the influence of the initial values in thesegmentation results for each analysed method has been per-formed. This study shows that both FGD as our method are lesssensitive to non-optimal choice of parameter values, getting simi-lar results with different parameter configurations.

We can conclude that the proposed method is a robust alterna-tive to known background modelling algorithms. The usage of thestochastic approximation framework allows to learn from onlinestreams of data in a natural way, so that it is particularly suitedfor this kind of application. Hence, we are confident that it will helpto build robust and accurate intelligent video surveillance systems.

Acknowledgments

This work has been partially supported by the Ministry ofScience and Innovation of Spain under Grant TIN2010-15351,project name ‘Probabilistic self organising models for the restora-tion of lossy compressed images and video’, and by Junta deAndalucía (Spain) under Contract TIC-01615, project name‘Intelligent Remote Sensing Systems’.

Appendix A. Stochastic approximation

Let Hi = (pi,li,Ci) be a vector comprising the parameters formixture component i, where i 2 {Back,Fore}. In what follows weare deriving the algorithm as if the covariance matrix CFore wereto be estimated. This is done for notational simplicity purposes,although CFore is never computed in practice. The derivation reliesin the methodology proposed in [40,41].

Let u(Hi,t) be an arbitrary function of Hi and the input samplet. Then we define the weighted mean of u(Hi,t) as:

huii ¼ E½PðijtÞuðHi; tÞ� ðA:1Þ

This allows expressing the conditional expectation of u(Hi,t) asfollows:

huiih1ii¼ E½PðijtÞuðHi; tÞ�

E½PðijtÞ� ¼R

pðtÞPðijtÞuðHi; tÞdtpi

¼Z

pðtÞPðijtÞpi

uðHi; tÞdt ¼Z

pðtjiÞuðHi; tÞdt

¼ E½uðHi; tÞji� ðA:2Þ

Therefore we can rewrite the mixture parameters in terms ofthe weighted means huii:

pi ¼ h1ii ðA:3Þ

li ¼htiih1ii

ðA:4Þ

Ci ¼hðt� liÞðt� liÞ

Tiih1ii

ðA:5Þ

If we have n samples (finite case), the linear least squaresapproximation for huii is:

huii ¼1n

Xn

j¼1

PðijtjÞuðHi; tjÞ ¼1n

Xn

j¼1

RjiuðHi; tjÞ ðA:6Þ

where we should remember that the posterior probability that mix-ture component i generated the sample tj is noted

Rji ¼ PðijtjÞ ðA:7Þ

As n ?1, the approximation of (A.10) converges to the exactvalue given by (A.1). Next we apply the Robbins–Monro stochasticapproximation algorithm (see Section 1.1 of [26]) to estimate iter-atively the value of the weighted means huii:

huiið0Þ ¼ Pðijt0ÞuðHi; t0Þ ðA:8ÞhuiiðnÞ ¼ huiiðn� 1Þ þ �ðPðijtnÞuðHi; tnÞ � huiiðn� 1ÞÞ ðA:9Þ

where � is the step size, which is a constant because the system istime varying (see Section 3.2 of [26]). Eq. (A.10) is more conve-niently written as:

huiiðnÞ ¼ ð1� �Þhuiiðn� 1Þ þ �RniuðHi; tnÞ ðA:10Þ

Now we derive an online stochastic approximation algorithmby applying Eq. (A.10) to equations (A.3)–(A.5). First we need todefine two auxiliary variables:

mi ¼ htii ðA:11ÞMi ¼ hðt� liÞðt� liÞ

Tii ðA:12Þ

The corresponding update equations are:

mðnÞ ¼ ð1� �Þmiðn� 1Þ þ �Rnitn ðA:13ÞMiðnÞ ¼ ð1� �ÞMiðn� 1Þ þ �Rniðtn � liðnÞÞðtn � liðnÞÞ

T ðA:14Þ

Then we are ready to rewrite (A.3)–(A.5):

piðnÞ ¼ ð1� �Þpiðn� 1Þ þ �Rni ðA:15Þ

liðnÞ ¼miðnÞpiðnÞ

ðA:16Þ

CiðnÞ ¼MiðnÞpiðnÞ

ðA:17Þ

Appendix B. Supplementary data

Supplementary data associated with this article can be found, inthe online version, at doi:10.1016/j.cviu.2011.01.007.

References

[1] W. Hu, T. Tan, L. Wang, S. Maybank, A survey on visual surveillance of objectmotion and behaviors, IEEE Transactions on Systems, Man, and Cybernetics,Part C: Applications and Reviews 34 (3) (2004) 334–352.

[2] T. Moeslund, A. Hilton, V. Kruger, A survey of advances in vision-based humanmotion capture and analysis, Computer Vision and Image Understanding 104(2006) 90–126.

[3] B. Morris, M. Trivedi, A survey of vision-based trajectory learning and analysisfor surveillance, IEEE Transactions on Circuits and Systems for VideoTechnology 18 (8) (2008) 1114–1127.

[4] M. Piccardi, Background subtraction techniques: a review, in: IEEEInternational Conference on Systems, Man and Cybernetics, 2004, pp. 3099–3104.

[5] L. Zappella, X. Lladó, J. Salvi, Motion segmentation: a review, in: Conference onArtificial Intelligence Research and Development, 2008, pp. 398–407.

[6] S.-C.S. Cheung, C. Kamath, Robust background subtraction with foregroundvalidation for urban traffic video, EURASIP Journal on Applied Signal Processing2005 (2005) 2330–2340.

[7] Y.-L. Tian, A. Hampapur, Robust salient motion detection with complexbackground for real-time video surveillance, in: IEEE Workshop on Motion andVideo Computing (WACV/MOTION), 2005, pp. 30–35.

[8] J. Barron, D. Fleet, S. Beauchemin, T. Burkitt, Performance of optical flowtechniques, in: IEEE Conference on Computer Vision and Pattern Recognition,CVPR, 1992, pp. 236–242.

[9] B. Galvin, B. McCane, K. Novins, D. Mason, S. Mills, Recovering motion fields: anevaluation of eight optical flow algorithms, in: British Machine VisionConference, BMVC, 1998, pp. 195–204.

[10] P. Spagnolo, T. Orazio, M. Leo, A. Distante, Moving object segmentation bybackground subtraction and temporal analysis, Image and Vision Computing24 (5) (2006) 411–423.

[11] B. Lo, S. Velastin, Automatic congestion detection system for undergroundplatforms, in: International Symposium on Intelligent Multimedia, Video andSpeech Processing, 2001, pp. 158–161.

[12] R. Cucchiara, M. Piccardi, P. Mello, Image analysis and rule-based reasoning fora traffic monitoring system, IEEE Transactions on Intelligent TransportationSystems 1 (2) (2000) 119–130.



[13] C. Ridder, O. Munkelt, H. Kirchner, Adaptive background estimation andforeground detection using Kalman-filtering, in: International Conference onRecent Advances in Mechatronics, 1995, pp. 193–199.

[14] D. Koller, J. Weber, T. Huang, J. Malik, G. Ogasawara, B. Rao, S. Russell, Towardsrobust automatic traffic scene analysis in real-time, in: InternationalConference on Pattern Recognition, ICPR, 1994, pp. 126–131.

[15] C. Wren, A. Azarbayejani, T. Darrell, A. Pentl, Pfinder: real-time tracking of thehuman body, IEEE Transactions on Pattern Analysis and Machine Intelligence19 (7) (1997) 780–785.

[16] C. Stauffer, W. Grimson, Learning patterns of activity using real-time tracking,IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (8) (2000)747–757.

[17] W. Grimson, C. Stauffer, R. Romano, L. Lee, Using adaptive tracking to classifyand monitor activities in a site, in: IEEE Conference on Computer Vision andPattern Recognition, CVPR, 1998, pp. 22–29.

[18] N. Friedman, S. Russell, Image segmentation in video sequences: aprobabilistic approach, in: Annual Conference on Uncertainty in ArtificialIntelligence, 1997, pp. 175–181.

[19] Z. Zivkovic, F. van der Heijden, Efficient adaptive density estimation per imagepixel for the task of background subtraction, Pattern Recognition Letters 27 (7)(2006) 773–780.

[20] D. Parks, S. Fels, Evaluation of background subtraction algorithms with post-processing, in: IEEE International Conference on Advanced Video and SignalBased Surveillance, AVSS, 2008, pp. 192–199.

[21] A. Elgammal, R. Duraiswami, D. Harwood, L. Davis, Background and foregroundmodeling using nonparametric kernel density estimation for visualsurveillance, in: IEEE Conference on Computer Vision and PatternRecognition, 2002, pp. 1151–1163.

[22] I. Haritaoglu, D. Harwood, L. Davis, W4: real-time surveillance of people andtheir activities, IEEE Transactions on Pattern Analysis and Machine Intelligence22 (8) (2000) 809–830.

[23] S. McKenna, S. Jabri, Z. Duric, A. Rosenfeld, H. Wechsler, Tracking groups ofpeople, Computer Vision and Image Understanding: CVIU 80 (1) (2000) 42–56.

[24] L. Li, W. Huang, I.Y.-H. Gu, Q. Tian, Statistical modeling of complexbackgrounds for foreground object detection, IEEE Transactions on ImageProcessing 13 (11) (2004) 1459–1472.

[25] K. Toyama, J. Krumm, B. Brumitt, B. Meyers, Wallflower: principles andpractice of background maintenance, in: IEEE International Conference onComputer Vision, ICCV, 1999, pp. 255–261.

[26] H.J. Kushner, G.G. Yin, Stochastic Approximation and Recursive Algorithms andApplications, Springer-Verlag, New York, NY, USA, 2003.

[27] T. Lai, Stochastic approximation, Annals of Statistics 31 (2) (2003) 391–406.[28] Z. Zivkovic, F. Van der Heijden, Recursive unsupervised learning of finite

mixture models, IEEE Transactions on Pattern Analysis and MachineIntelligence 26 (5) (2004) 651–656.

[29] Y. Pang, Y. Yuan, X. Li, Iterative subspace analysis based on feature linedistance, IEEE Transactions on Image Processing 18 (4) (2009) 903–907.

[30] C. Tsao, Y.-C. Chang, A stochastic approximation view of boosting,Computational Statistics and Data Analysis 52 (1) (2007) 325–334.

[31] S. Gadat, L. Younes, A stochastic algorithm for feature selection in patternrecognition, Journal of Machine Learning Research 8 (2007) 509–547.

[32] Y. Wei, S. Gelfand, J. Krogmeier, Noise-constrained least meansquares algorithm, IEEE Transactions on Signal Processing 49 (9) (2001)1961–1970.

[33] E. Chong, I.-J. Wang, S. Kulkarni, Noise conditions for prespecified convergencerates of stochastic approximation algorithms, IEEE Transactions onInformation Theory 45 (2) (1999) 810–814.

[34] E. López-Rubio, Probabilistic self-organizing maps for continuous data, IEEETransactions on Neural Networks 21 (10) (2010) 1543–1554.

[35] E. López-Rubio, Probabilistic self-organizing maps for qualitative data, NeuralNetworks 23 (10) (2010) 1208–1225.

[36] H. Robbins, S. Monro, A stochastic approximation method, Annals ofMathematical Statistics 22 (3) (1951) 400–407.

[37] B. Delyon, M. Lavielle, E. Moulines, Convergence of a stochastic approximationversion of the EM algorithm, Annals of Statistics 27 (1) (1999) 94–128.

[38] M.-A. Sato, S. Ishii, On-line em algorithm for the normalized Gaussian network,Neural Computation 12 (2) (2000) 407–432.

[39] N. Bouguila, D. Ziou, A hybrid SEM algorithm for high-dimensionalunsupervised learning using a finite generalized Dirichlet mixture, IEEETransactions on Image Processing 15 (9) (2006) 2657–2668.

[40] E. López-Rubio, J.M. Ortiz-de-Lazcano-Lobato, D. López-Rodrfguez,Probabilistic PCA self-organizing maps, IEEE Transactions on NeuralNetworks 20 (9) (2009) 1474–1489.

[41] E. López-Rubio, Multivariate Student-t self-organizing maps, Neural Networks22 (10) (2009) 1432–1447.

[42] S. Klein, M. Staring, J. Pluim, Evaluation of optimization methods for nonrigidmedical image registration using mutual information and b-splines, IEEETransactions on Image Processing 16 (12) (2007) 2879–2890.

[43] S. Klein, J. Pluim, M. Staring, M. Viergever, Adaptive stochastic gradient descentoptimisation for image registration, International Journal of Computer Vision81 (3) (2009) 227–239.

[44] Y. Trivedi, L. Kurz, A class of robust image processors, Pattern Recognition 27(8) (1994) 1111–1125.

[45] A. Chowdhury, R. Chellappa, Stochastic approximation and rate-distortionanalysis for robust structure and motion estimation, International Journal ofComputer Vision 55 (1) (2003) 27–53.

[46] H. Takeda, S. Farsiu, P. Milanfar, Kernel regression for image processing andreconstruction, IEEE Transactions on Image Processing 16 (2) (2007) 349–366.

[47] E. López-Rubio, Restoration of images corrupted by Gaussian and uniformimpulsive noise, Pattern Recognition 43 (5) (2010) 1835–1846.

[48] S. Ando, Image field categorization and edge/corner detection from gradientcovariance, IEEE Transactions on Pattern Analysis and Machine Intelligence 22(2) (2000) 179–190.

[49] D. Hall, J. Nascimento, P. Ribeiro, E. Andrade, M.P., Comparison of targetdetection algorithms using adaptive background models, in: IEEE InternationalWorkshop on Visual Surveillance and Performance Evaluation of Tracking andSurveillance (VS-PETS), 2005, pp. 113–120.

[50] T. Schlogl, C. Beleznai, M. Winter, H. Bischof, Performance evaluation metricsfor motion detection and tracking, in: International Conference on PatternRecognition, ICPR, vol. 4, 2004, pp. 519–522.

[51] J. Black, T. Ellis, P. Rosin, A novel method for video tracking performanceevaluation, in: IEEE International Workshop on Visual Surveillance andPerformance Evaluation of Tracking and Surveillance (VS-PETS), 2003, pp.125–132.

[52] J. Nascimento, J. Marques, Performance evaluation of object detectionalgorithms for video surveillance, IEEE Transactions on Multimedia 8 (4)(2006) 761–774.

[53] N. Wollborn, R. Mech, Procedure for objective evaluation of vop generationalgorithm, MPEG Committee, Document ISO/IEC/JTC1/SC29/WG11 M270.

[54] P. Villegas, X. Marichal, Perceptually-weighted evaluation criteria forsegmentation masks in video sequences, IEEE Transactions on ImageProcessing 13 (8) (2004) 1092–1103.

[55] W. Zhou, R. Serfling, Multivariate spatial U-quantiles: a Bahadur–Kieferrepresentation, a Theil–Sen estimator for multiple regression, and a robustdispersion estimator, Journal of Statistical Planning and Inference 138 (6)(2008) 1660–1678.

[56] R. Serfling, Nonparametric multivariate descriptive measures based on spatialquantiles, Journal of Statistical Planning and Inference 123 (2) (2004) 259–278.

[57] H.P. Lopuhaa, P.J. Rousseeuw, Breakdown points of affine equivariantestimators of multivariate location and covariance matrices, The Annals ofStatistics 19 (1) (1991) 229–248.

[58] E. Dubois, The structure and properties of color spaces and the representationof color images, Synthesis Lectures on Image, Video, and MultimediaProcessing 11 (2009) 1–129.

[59] M. Sato, S. Ishii, On-line EM algorithm for the normalized Gaussian network,Neural Computation 12 (2) (2000) 407–432.

[60] P. KaewTrakulPong, R. Bowden, A real time adaptive visual surveillance systemfor tracking low-resolution colour targets in dynamically changing scenes,Image and Vision Computing 21 (9) (2003) 913–929.

Documents

Stochastic approximation for background modelling