LOCAL BACKGROUND SPRITE GENERATION

Andreas Krutz1, Alexander Glantz1, Michael Frater2, and Thomas Sikora1

1Communication Systems Group, Technische Universitat Berlin,EN-1, Einsteinufer 17, 10587 Berlin, Germany, {krutz,glantz,sikora}@nue.tu-berlin.de

2 School of Information Technology and Electrical Engineering, University of New South Wales,Canberra ACT 2600, Australia, m.frater@adfa.edu.au

ABSTRACT

Background modeling of video sequences can be used inmany different applications. For video object segmenta-tion it is often applied in a background subtraction method.When conventional sprites like single or multiple spritesare used a background sequence has to be reconstructedfrom the model. The double mapping into the coordinatesystem of the sprite and back can lead to severe distortionof the background model and therefore to erroneous seg-mentation masks. We present a novel background model-ing approach that lessens distortion. These so-called localbackground sprites are built for every reference frame in-dependently and fit its original size. Experimental resultsshow that this new approach clearly outperforms conven-tional background sprites in terms of PSNR.

1. INTRODUCTION

Background modeling – meaning the description of thebackground of a video sequence – is an important researchfield. Application scenarios range from video surveillancesystems to video coding.

A so-called background sprite or single sprite gener-ated over a certain number of frames can be such a back-ground model. A background sprite is an image that typ-ically is of large size and contains only the backgroundpixels of a video sequence. This approach can be extendedto so-called multiple sprites or super-resolution sprites. Incase of multiple sprites the video sequence is divided intoparts and for each part one background sprite is generatedindependently. In case of super-resolution a higher qualitybackground sprite is generated for a sequence.

The potential for using background sprites for object-based video coding has been summarized in [1]. Fur-thermore, background modeling is an efficient means forvideo object segmentation. Various approaches using sin-gle or multiple background sprites in a background sub-traction method have demonstrated that this kind of back-ground model is very promising [2], [3]. However, themapping of pixel content from various frames in a sceneinto a single sprite or a collection of multiple sprites maycause severe geometrical distorion of the background. Forreconstruction of the background of a single frame a sec-ond mapping needs to be performed which causes addi-tional distortion. Object segmentation is one application

Figure 1. Single sprite, sequence “Stefan”, referenceframe 253

that is degraded by such distortion.In our proposed local background sprite algorithm a

mapping of content from many frames in a scene is per-formed for each individual frame for background construc-tion. In other words, global motion estimation (registra-tion) is performed from many adjacent frames into theframe where the background needs to be reconstructed.No backward mapping is required. Thus, our backgroundsprites are local and there are as many individual spritesgenerated as frames exist in a sequence. This will resultin a more precise background reconstruction compared toconventional global sprites.

This paper is organized as follows. Section 2 providesa short introduction to conventional background sprite tech-niques. Section 3 describes the new background modelingapproach. The experimental evaluation in Section 4 showsthe excellent performance of this new approach and Sec-tion 5 concludes the paper.

2. GENERAL BACKGROUND SPRITES

2.1. Single Sprites

A single sprite models the background of a given sequencein one single image. This image usually is of large sizeand contains only the pixels from the background of thesequence. An example of a single sprite is depicted inFigure 1.

For the creation of a single sprite a reference frame ischosen and all other frames of the sequence are warpedinto the coordinate system of the reference. For that long-term higher-order motion parameters are computed thatdescribe this transformation [4]. First, short-term parame-ters are calculated using global motion estimation (GME)with the well-known 8-parameter motion model. Theseshort-term parameters are then accumulated by simple ma-trix multiplication as shown in Figure 2.

Wref−2,ref = Wref−2,ref−1 ·Wref−1,ref

Wref−3,ref = Wref−3,ref−2 ·Wref−2,ref

Wref+2,ref = Wref+2,ref+1 ·Wref+1,ref

Wref+3,ref = Wref+3,ref+2 ·Wref+2,ref

......

Wref−3,ref

Wref−2,ref

Wref+3,ref

Wref+2,ref

Iref−3 Iref−1 Iref Iref+1 Iref+2Iref−2 Iref+3· · · · · ·

Wref−3,ref−2 Wref−2,ref−1 Wref−1,ref Wref+1,ref Wref+2,ref+1 Wref+3,ref+2

Figure 2. Creation of long-term motion parameters

(a) Frames 0-244

(b) Frames 245-261 (c) Frames 262-297

Figure 3. Multiple sprites, sequence “Stefan”

By transforming all frames using the long-term pa-rameters into the coordinate system of the backgroundsprite a stack of size M × N × S is created where Mand N are the dimensions of the final background spriteand S is the number of frames in the sequence. The im-ages in this stack are then blended together to generate thebackground sprite. The intention is to eliminate the fore-ground objects in the background sprite. Blending filtersnormally used are the mean or the median of all pixel val-ues lying in dimension S on top of each other.

2.2. Multiple Sprites

Depending on the camera motion the generation of a sin-gle sprite leads to severe distortion in the border area.These cause decreased quality of the background modeland additionally increase coding costs which can be seenin Figure 1. In some cases where the camera pan exceedsan angle of 90 degrees generation of one single sprite isnot even possible. These drawbacks have lead to so-calledmultiple sprites [2], [5].

For the generation of multiple sprites the camera mo-tion is analyzed first. The next step is to split the video se-

quence into parts depending on the analyzed motion. Af-terwards a background sprite is generated independentlyfor every part as shown previously. An example of a mul-tiple sprite is shown in Figure 3.

2.3. Super-resolution Sprites

Super-resolution is a technique that aims on increasing thequality of an image. A high-resolution counterpart is builtfrom several images with lower resolution. These imagescan originate from one camera taking multiple images ofa scene in time, from multiple cameras each taking oneimage in time or from the frames of a moving video cam-era. The idea of this approach is that an arbitrary point isvisible several times.

This method can easily be extended to backgroundsprite generation. When building a background sprite avideo sequence is used. After global motion estimationand transformation into the coordinate system of the ref-erence frame the pixel locations are rarely integer values.This feature can be used to generate a background spriteof higher resolution. For more details see [6] or [7].

3. LOCAL BACKGROUND SPRITES

A local background sprite specifies a model of the back-ground. Other than general background sprites one modelis built for every frame and not one model for the wholevideo sequence. Only the local temporal neighborhood ofeach reference frame is taken into account for sprite gener-ation. The dimensions of a local background sprite matchthose of the reference frame. The idea is to minimize dis-tortion in background regions. When a background frameis reconstructed from a general background sprite distor-tion can be severe. This is due to accumulated errors in theglobal motion estimation, non-ideal interpolation and thedouble mapping into the coordinate system of the back-ground sprite and back.

The algorithm for modeling local background spritesfor a given video sequence is depicted in Figure 4. Its

GME

Blendingn=1

ref=ref+1

Warp frame

to reference frame

Iref−n

Sequence of local

background sprites

Warp frame

to reference frame

Iref+n

ref < ref max

Yes

No

Quality

sufficient?

No

Yes

Video n=1, ref=0

n=n+1

Figure 4. Modeling the background by means of local background sprites

different parts are explained in this section.

3.1. Global Motion Estimation

For global motion estimation a hierarchical gradient de-scent approach based on the Gauss-Newton method is ap-plied as used in [8] or [9]. A block chart of the approachcan be seen in Figure 5. The algorithm estimates the dis-placement between two temporally adjacent frames Ip andIq of a sequence using the 8-parametric higher-order per-spective motion model which is described by the follow-ing equations

xq =m0xp + m1yp + m2

m6xp + m7yp + 1(1)

yq =m3xp + m4yp + m5

m6xp + m7yp + 1(2)

where (xp yp)T is the location of a pixel in frame Ip and(xq yq)T its corresponding position in frame Iq . The pa-rameters m0 to m7 describe the motion by means of trans-lation, scaling, rotation, sheering and perspective transfor-mation.

The algorithm first generates a 4-step image pyramidfor the two frames to register. The image pyramid con-tains the original frames, two downsampled versions andone in the upsampled domain. For downsampling a 5-tapLe-Gall wavelet filter is used and for upsampling a 7-tapDaubechies wavelet filter. The first gradient descent stepis performed on the coarsest resolution and is initializedwith a translational motion model using the feature trackerpresented by Kanade et al. [10]. The algorithm then per-forms a gradient descent step in every other layer of theimage pyramid using the motion parameters from the stepbefore as initialization.

Since the short-term displacement between two framesIp and Iq is used several times while creating all localbackground sprites for a video sequence the motion pa-rameters are computed in a preprocessing step. This meansfor a sequence with n frames the set T of transformationmatrices

T = {W0,1,W1,2, . . . ,Wn−2,n−1} (3)

and its inverted correspondences

Tinv = {W−10,1,W

−11,2, . . . ,W

−1n−2,n−1} (4)

are computed where |T | = |Tinv| = n−1, W−1p,q = Wq,p

and

Wp,q =

m0,p,q m1,p,q m2,p,q

m3,p,q m4,p,q m5,p,q

m6,p,q m7,p,q 1

(5)

is the transformation matrix between frames Ip and Iq .

3.2. Warping and Blending

For every reference frame a local background sprite isbuilt. The algorithm iteratively transforms neighboringframes into the coordinate system of the reference. Thisproduces a dynamically growing image stack of size M ×N × St where M and N are the dimensions of the refer-ence frame and St = 2t + 1 is the depth of the stack instep t. In step t = 0 the stack only contains the referenceframe. This approach can be seen in Figure 6.

For the transformation of an arbitrary frame into thereference’s coordinate system the short-term motion pa-rameters from the preprocessing step are accumulated togenerate long-term parameters which can be seen in Fig-ure 2. The global motion estimation can only computethe displacement between two frames by approximation.Due to existing small errors and the accumulation the er-ror in the long-term parameters grows larger with increas-ing temporal distance to the reference frame. Hence, thelong-term parameters are used as initialization for anothergradient descent step to reduce this error.

In every step t the images in the stack are merged to-gether to build a preliminary local background sprite ofsize M ×N . For this purpose a so-called blending filter isused which here is a median filter. The median returns themiddle value of an ordered dataset – in this case a set ofluminance and chrominance values respectively. The ad-vantage over using a mean filter is its robustness for out-liers. Additionally, the median is always an element of theset itself and does not produce new values.

By successively adding temporally neighboring framesthe foreground objects in the preliminary local backgroundsprites are removed step by step. This is due to the areabehind the foreground objects that is disclosed because oftheir movements. This can be seen in Figure 7. Depictedare the preliminary local background sprites for the “Ste-fan” sequence for various steps t. One can clearly see that

24 2

Gauss−Newton

Affine model

Gauss−Newton

Persp. model

Gauss−Newton

Persp. model

Gauss−Newton

Persp. model

q

Frames

and p

I I

Init

Params

Init

Params

Init

Params

Init

Params

m2

m6 m8 m8 m8

m8 m8 m8

m8

Feature

tracking

Figure 5. Global motion estimation algorithm

I228

I229 I231

I232

I230

I’229

I’228

I’231

I230

I’232

...

...

x

y

z

Figure 6. Creation of an image stack for the generation of a local background sprite, sequence “Stefan”, reference frame230

the foreground object has nearly completely vanished af-ter eight blending steps.

It is possible to evaluate the quality of the backgroundmodel in every step subjectively. However, an automaticevaluation criterion is desirable that stops the generationof the local background sprite when its quality is goodenough and the foreground objects are removed suffic-iently. An approach for automatic quality evaluation ispresented next.

3.3. Quality Evaluation of Local Background Sprites

A possible measure for the difference between two im-ages or frames is the root mean square error (RMSE). TheRMSE between a reference frame Iref (x, y) and its pre-liminary local background sprite Ibs,t(x, y) in step t is de-fined by

RMSEt =

vuut 1

MN

M−1Xi=0

N−1Xj=0

(Iref (i, j)− Ibs,t(i, j))2 (6)

where M and N are the dimensions of the reference frameand the preliminary local background sprite respectively.The preliminary local background sprite in step t = 0is the reference frame itself so that Ibs,t=0 = Iref andRMSEt=0 = 0. Since the foreground objects vanish stepby step the RMSE value increases successively. There-for, the difference of the RMSE values in two consecutivesteps

∆RMSEt = RMSEt −RMSEt−1 (7)

(a) Reference frame (b) Preliminary local backgroundsprite, t = 2

Figure 8. Partitioning of images into blocks of size 25 ×25, sequence “Stefan”, reference frame 230

decreases. When the foreground objects are completelyeliminated, the values RMSEt and ∆RMSEt changeonly marginally.

At the beginning, foreground objects are still presentin the preliminary local background sprite. Change inthese areas leads to high values of ∆RMSEt. After sev-eral steps, most of the foreground is eliminated whichleads to lower values of ∆RMSEt. It holds true that

∆RMSEa >= ∆RMSEb

for a <= b. The value ∆RMSEt can be interpreted asa measure for the information about the background thathas been added to the local background sprite in step t.

Using the measure ∆RMSEt is not without problemswhen only small foreground objects are present. Smallobjects only take a minor percentage of the whole frame.

(a) Step t = 1, St = 3 (b) Step t = 2, St = 5

(c) Step t = 3, St = 7 (d) Step t = 8, St = 17

Figure 7. Preliminary local background sprites, sequence “Stefan”, reference frame 230

The influence of errors in these areas on the measure issmall compared to the sum of errors in the backgroundregions. In this case plotting RMSEt and ∆RMSEt

against time produces very flat curves which make a de-cision on the quality of the preliminary local backgroundsprite very difficult. Therefore, we define matrices con-taining the blockwise calculated value ∆RMSEt, whichwe call dRMSE-matrices. Reference frame and prelim-inary local background sprite are divided into blocks offixed size which can be seen in Figure 8. Within this work,various block sizes between 10×10 and 40×40 have beentested. The problem with small block sizes is that the pro-file of the dRMSE-matrices is of high frequency. Withlarge block sizes the unwanted effect of averaging of largeregions sets in again. In this work a fixed block size of25× 25 is used. The value RMSEt is then calculated forevery block independently. Therefore, no averaging overthe whole frame takes place. Distinct areas in the prelim-inary local background sprite can be evaluated indepen-dently. Furthermore, the difference to the block values inthe step before is computed using Equation 7.

Figure 9 shows the corresponding matrices for the ex-ample in Figure 7. At the beginning the plot of the matrixis very wavy. With increasing steps t the matrices flat-ten successively. This means, the more temporally neigh-boring frames are transformed into the local background

sprite’s coordinate system the less information about thebackground of the reference frame is gained. The peakin the middle of the matrices in Figure 9 results from themoving tennis player and the area behind him that is dis-closed. However, the RMSE in the background regionsdoesn’t change significantly. After 8 blending steps (Fig-ure 9(d)) – 16 frames and the reference frame have beenblended together – the matrix is nearly flat in all regions.This corresponds with the results in Figure 7.

The quality of the preliminary local background spritesnow is assessable in a very differentiated way. Assum-ing the generation of the local background sprite is tobe aborted when there is no more information added inany region, meaning the matrix presented is flat in everyregion. We present three possible evaluation criteria fordRMSE-matrices.

The easiest way to evaluate the matrices is their max-imum value. The maximum provides information aboutthe biggest change of a block in the preliminary local back-ground sprite. The curve of the maximum plotted againststep t can be seen in Figure 10(a).

Another way to evaluate the matrices is the varianceof their values. The variance provides information aboutthe distribution of values in a dRMSE-matrix. The curveof the variance plotted against step t can be seen in Figure10(b).

0

5

10

15

0

2

4

6

8

100

5

10

15

20

xy

dRM

SE

(a) Step t = 1

0

5

10

15

0

2

4

6

8

100

5

10

15

20

xy

dRM

SE

(b) Step t = 2

0

5

10

15

0

2

4

6

8

100

5

10

15

20

xy

dRM

SE

(c) Step t = 3

0

5

10

15

0

2

4

6

8

100

5

10

15

20

xy

dRM

SE

(d) Step t = 8

Figure 9. dRMSE-matrices using blocks of size 25× 25, sequence “Stefan”, reference frame 230

The last way to evaluate the matrices is inspired bythe window-based feature tracker of Kanade et al. [10]. Intheir work, they define that a good feature is one that canbe tracked well. This is the case when its texturizing ishigh. Therefore, the gradient∇g = (gx gy)T is generatedfor each possible window. The gradients then are used tocreate a symmetric 2× 2 matrix

Z = ∇g∇gT =[

g2x gxgy

gxgy g2y

](8)

The eigenvalues of matrix Z are related to the texture.Two small eigenvalues mean a nearly homogeneous in-tensity profile. A large and a small eigenvalue correspondto a unidirectional texture pattern. Two large eigenvaluesrepresent any highly textured pattern that can be trackedwell. The dRMSE-matrices can be compared to the in-tensity profile of a feature window. On the contrary, weare looking for a nearly constant profile which means twosmall eigenvalues. Therefore, in every step t matrix Zis computed from the gradientes of the dRMSE-matrices.The curves of the eigenvalues plotted against step t can beseen in Figure 10(c).

For every evaluation criterion used we assume localbackground sprite quality sufficient when maximum, vari-

Evaluation criterion ThresholdMaximum 5.0Variance 0.2Eigenvalues 5.0

Table 1. Thresholds for evaluation criteria

ance and eigenvalues respectively lie below their prede-fined threshold values. The used thresholds can be see inTable 1. Outliers – step t = 4 in the examples in Figure 10– come from sudden movements of the foreground objectsor changes in camera speed.

4. EXPERIMENTAL EVALUATION

We have evaluated our approach using four test sequences.The first sequence is called “Allstars (Shot 1)” (352×288,250 frames) and is recorded from a soccer broadcast ofgerman television. It mainly consists of translational mo-tion but is difficult to handle for global motion estimationbecause of its low-frequency content (green pitch). Thesecond sequence is called “Mountain” (352 × 192, 100frames) and is part of a BBC documentation. It showsa leopard chasing an animal while moving down a hill.

1 2 3 4 5 6 7 8 9 100

20

40

60

80

100

120

step t

max

imum

dR

MS

E

(a) Maximum

1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

9

10

step t

varia

nce

dRM

SE

(b) Variance

1 2 3 4 5 6 7 8 9 100

50

100

150

200

250

300

350

400

450

500

step t

eige

nval

ues

dRM

SE

Eigenvalue 1Eigenvalue 2

(c) Eigenvalues

Figure 10. Various evaluation critera for dRMSE-matrices, sequence “Stefan”, reference frame 230

This sequence consists also mainly of translational motionbut has a high-frequency background (mountains). Thethird sequence is called “Race1 (View 0)” (544 × 336,100 frames) and is part of an MPEG testset for multiviewsequences. It shows a kart race. The forth sequence is thewell-known “Stefan” sequence (352 × 240, 300 frames).Both the content and the motion of the sequence are verycomplex. It consists of low-frequency (tennis court) andhigh-frequency parts (audience). The camera motion iscomposed of translation, scaling and perspective transfor-mation.

For evaluation of background quality we compute thebackground PSNR of the model using equation

PSNR = 10 · log10

(2552

MSE

)(9)

for the sequence’s luminance part. We have generatedground truth masks of the foreground objects for all se-quences to compute the PSNR only between the back-ground pixels of the original reference frame and the back-ground model.

First, we compare all possible evaluation criteria, i.e.maximum, variance and eigenvalues, for the generationof local background sprites. Figure 11 shows the back-ground PSNR for the background models generated. Ta-ble 2 shows the mean PSNR values. One can clearly seethat for every sequence the maximum criterion producesthe best background quality (bold values in Table 2). Thisis due to the fact that local background sprite generationstops using the least temporally neighboring frames whenthe maximum criterion is chosen.

The high values in the last quarter of sequence “All-stars (Shot 1)” (Figure 11(a)) come from the camera that isstatic in that part of the sequence. The translational initial-ization of the global motion estimation algorithm is veryexact and the gradient descent algorithm produces verysmall errors.

The PSNR profile of sequence “Mountain” in Figure11(b) is almost constant. Except for low values at the be-ginning and at the end of the sequence. These are explain-able with errors in registration.

The low values around frame 30 of sequence “Race1(View 0)” (Figure 11(c)) come from a fast camera pan that

produces big translational motion. Contrary to the last partof sequence “Allstars (Shot 1)” the translational initializa-tion in that part of the sequence produces higher errors forthe global motion estimation.

The peaks around frames 25, 50 and 120 and the dropin the last 30 frames of sequence “Stefan” (Figure 11(d))have similar causes. The camera motion is static or nearlystatic in parts with peaks and moves very fast at the end ofthe sequence.

Next, the local background sprites are compared tocommon background models. We use on one hand thelocal background sprites provided by the worst criterion –which in all cases is variance, except for sequence “Race1(View 0)” which is eigenvalues. On the other hand we usereconstructed background sequences from single, multipleand super-resolution sprites repspectively. The plots forthe comparison can be seen in Figure 12. Table 3 showsthe mean PSNR values. One can clearly see that the newapproach outperforms common background models evenwhen the worst evaluation criterion is used at about 2− 6dB (bold values in Table 3).

For sequence “Stefan” (Figure 12(d)) only the first 250frames are taken into account for the mean PSNR valueas the super-resolution sprite existed only for that range.The three peaks in the curve for multiple sprites resultfrom the generation process that adds the original pixelsat the position of the reference frame in the backgroundsprite. Therefore, the quality at the three reference framesis nearly optimal.

5. CONCLUSION

We have presented a novel approach for background mod-eling of video sequences. Conventional background spritesmodel the background of the sequence in one image thatusually is of large dimensions. Other than that local back-ground sprites are generated for each reference frame in-dividually and are of the same size as the reference. Whenusing conventional background sprites for applications likeimage segmentation the frames containing the backgroundinformation have to be reconstructed from the sprite im-age. This step is not necessary when using local back-ground sprites. Therefore, distortion is reduced. It hasbeen shown that the quality of the background model us-

Sequence Algorithm PSNR [dB]Allstars (Shot 1) Local background sprite (maximum) 34.4345

Local background sprite (variance) 33.7043Local background sprite (eigenvalues) 33.8240

Mountain Local background sprite (maximum) 35.1592Local background sprite (variance) 34.7299Local background sprite (eigenvalues) 34.8040

Race1 (View 0) Local background sprite (maximum) 34.6666Local background sprite (variance) 33.9092Local background sprite (eigenvalues) 33.7659

Stefan Local background sprite (maximum) 29.5146Local background sprite (variance) 29.0908Local background sprite (eigenvalues) 29.2079

Table 2. Mean background PSNR values for local background sprites generated using all evaluation criteria

Sequence Algorithm PSNR [dB]Allstars (Shot 1) Single sprite 29.0938

Local background sprite (variance) 33.7043Mountain Single sprite 28.8877

Local background sprite (variance) 34.7299Race1 (View 0) Single sprite 30.1350

Local background sprite (eigenvalues) 33.7659Stefan (1-250) Multiple sprites 24.7312

Super-resolution sprite 27.4404Local background sprite (variance) 29.4479

Table 3. Mean background PSNR comparing reconstructed single/multiple/super-resolution sprites with local backgroundsprites (worst evaluation criterion)

0 50 100 150 200 25030

35

40

45

50

55

frame

back

grou

nd P

SN

R [d

B]

MaximumVarianceEigenvalues

(a) Sequence “Allstars (Shot 1)”

0 10 20 30 40 50 60 70 80 90 10028

30

32

34

36

38

40

frame

back

grou

nd P

SN

R [d

B]

MaximumVarianceEigenvalues

(b) Sequence “Mountain”

0 10 20 30 40 50 60 70 80 90 10029

30

31

32

33

34

35

36

37

38

39

back

grou

nd P

SN

R [d

B]

frame

MaximumVarianceEigenvalues

(c) Sequence “Race1 (View 0)”

0 50 100 150 200 250 30022

24

26

28

30

32

34

36

frame

back

grou

nd P

SN

R [d

B]

MaximumVarianceEigenvalues

(d) Sequence “Stefan”

Figure 11. Background PSNR of local background sprites using various evaluation criteria for dRMSE-matrices

ing this new technique clearly outperforms conventionalbackground models.

Further work is required concerning all fixed thresh-olds used in this work. The block sizes of the dRMSE-matrices can be adaptively adjusted using a preliminarysegmentation that provides information about the meanobject size in the reference frame. The threshold valuesfor the evaluation criteria can also be adaptively adjusted.The curves are similar to an exponential path. Curve fit-ting techniques can estimate the path using past values andadjust the threshold depending on a possible convergence.

References[1] T. Sikora, “Trends and perspectives in image and

video coding,” Proceedings of the IEEE, vol. 93, pp.6–17, January 2005.

[2] D. Farin, P. H. N. de With, and W. Effelsberg, “Videoobject segmentation using multi-sprite backgroundsubtraction,” in Int. Conf. on Multimedia and Expo(ICME), Taipei, Taiwan, June 2004.

[3] A. Krutz, M. Kunter, M. Mandal, M. Frater, andT. Sikora, “Motion-based object segmentation usingsprites and anisotropic diffusion,” in 8th Interna-tional Workshop on Image Analysis for MultimediaInteractive Services (WIAMIS), Santorini, Greece,June 2007.

[4] A. Smolic, T. Sikora, and J.-R. Ohm, “Long-term global motion estimation and its applicationfor sprite coding,” IEEE Transactions on Circuitsand Systems for Video Technology, vol. 9, no. 8, pp.1227–1242, December 1998.

[5] M. Kunter, A. Krutz, M. Droese, M. Frater, andT. Sikora, “Object-based multiple sprite codingof unsegmented videos using H.264/AVC,” inIEEE International Conference on Image Process-ing (ICIP’07), San Antonio, USA, Sept. 2007.

[6] G. Ye, M. Pickering, M. Frater, and J. Arnold, “A ro-bust approach to super-resolution sprite generation,”in Int. Conf. on Image Processing (ICIP’05), Gen-ova, Italy, Sept. 2000.

0 50 100 150 200 25025

30

35

40

45

50

frame

back

grou

nd P

SN

R [d

B]

Single spriteLocal background sprite (variance)

(a) Sequence “Allstars (Shot 1)”

0 10 20 30 40 50 60 70 80 90 10020

22

24

26

28

30

32

34

36

38

frame

back

grou

nd P

SN

R [d

B]

Single spriteLocal background sprite (variance)

(b) Sequence “Mountain”

0 10 20 30 40 50 60 70 80 90 10020

22

24

26

28

30

32

34

36

38

Single spriteLocal background sprite (eigenvalues)

(c) Sequence “Race1 (View 0)”

0 50 100 150 200 250 30015

20

25

30

35

40

45

frame

back

grou

nd P

SN

R [d

B]

Multiple spritesSuper−resolution spriteLocal background sprite (variance)

(d) Sequence “Stefan”

Figure 12. Comparing background PSNR of reconstructed single/multiple/super-resolution sprite with local backgroundsprite (worst evaluation criterion)

[7] M. Kunter, J. Kim, and T. Sikora, “Super-resolutionmosaicing using embedded hybrid recursive flow-based segmentation,” in IEEE Int. Conf. on In-formation, Communication and Signal Processing(ICICS’05), Bangkok, Thailand, Dec. 2005.

[8] A. Krutz, M. Frater, and T. Sikora, “Improved imageregistration using the up-sampled domain,” in Int.Conf. on Multimedia Signal Processing (MMSP’06),Victoria, Canada, Oct. 2006.

[9] Frederic Dufaux and Janusz Konrad, “Efficient, ro-bust, and fast global motion estimation for videocoding,” IEEE Transactions on Image Processing,vol. 9, pp. 497–501, 2000.

[10] S. Baker and I. Matthews, “Lucas-kanade 20 yearson: A unifying framework,” International Journal ofComputer Vision, vol. 56, pp. 221–255, Feb. 2004.