Color Image Segmentation of Foreground and Background ......understand the object, it is necessary to segment it from its image. The segmentation of foreground and background also

Color Image Segmentation of Foreground and Background based on Mean Shift Algorithm

Liang jingmin

Guangdong Women’s Polytechnic College, China [email protected]

Abstract

A new color image segmentation algorithm of foreground and background based on Mean Shift has been proposed and implemented. The image is pre-segmentation in some areas using Mean Shift

algorithm， under the MAP--MRF framework, we can get a source--sink graph about the image, and then making use of the color histogram, we can give different weights to every region. Use max-- flow algorithm to segment the weighted graph and a min--cut can be got. According to the users' inputs, some boundary can be processed by matting algorithm. The experimental results show that the method

only requires less users' inputs and has a better segmentation efficiency．

Keywords: Segmentation Of Foreground And Background, Mean Shift, Color Histogram, Max-Flow Min-Cut

1. Introduction

In all images, we are often interested in some part, which has specific statistics. To analysis and understand the object, it is necessary to segment it from its image. The segmentation of foreground and background also belongs to image segmentation, some one call this digital matting, The matting problem is to extract a foreground element of arbitrary shape from a background image by estimating a color and opacity for the foreground element at each pixel. It has been widely used in image processing,

film and video production and virtual reality． The existing image segmentation algorithm of foreground and background for dealing with a single

background or transparent digital matting effect has been very good. But even so, it also requires two different color background scene photographs can completely determine the edge information [1]. In the field of natural matting, A. Berman proposed a Knockout method in 2000 [2, 3]. In addition to the current natural matting boundary region marked by hand initially, the main statistical methods are used as compensation. Principal component analysis [4, 5], Bayesian method [6, 7] and matting methods based on HVS method [8] are based on statistical representative. In 2004, Jian Sun et al. of Microsoft Research Asia proposed Poisson matting [10] according to Poisson Editing method proposed by Patrick Perez et al. in 2003 [9]. In fact, the transparency is as an intrinsic property of the image--field obtained by solving the Poisson equation to get value of the unknown region. This method looked the transparency of the pixel value as the interrelated overall, which created a new era of digital matting based on transparency. However, for the high clarity and complex background or the similar case of the background color of the edge, the processing effect of the above digital matting method is still not ideal.

First, the image is pre-segmentation in some regions using Mean Shift algorithm under the MAP--MRF framework, and we can get a source--sink graph about the image, and then making use of the color histogram, we can give different weights for every regions. Use max flow algorithm to segment the weighted graph and a min--cut can be gotten. According to the users inputs, some boundary can be processed by matting algorithm.

Color Image Segmentation of Foreground and Background based on Mean Shift Algorithm Liang jingmin

International Journal of Advancements in Computing Technology(IJACT) Volume4, Number1, January 2012 doi: 10.4156/ijact.vol4.issue1.14

127

2. Methodology 2.1 Mean Shift Theory

Mean Shift is a non-parametric method of kernel estimation method. It is the nature of the use of

nuclear function without the need of the whole region to estimate the probability density, and uses the window--point x of density gradient estimate and the mean migration vector as a nuclear export window moving step, maximum density is achieved by iterative search. Mean Shift method in practical applications has high efficiency and good stability.

The direction of maximum density for Mean Shift vector is called vector:

])||(||

)||(||[)(

1

2

1

2

x

h

xxg

h

xxgx

xmhn

i

i

n

i

ii

(1)

To better understand the meaning of Mean Shift vector, we may assume 1)( xg , and the weight

ix is set to 1/n so that the Mean Shift vector can be written as:

n

ii xx

nxmh

1

)(1

)( (2)

Mean Shift vector will make the sample points move toward the most intensive sample points. The

weight function is constant. In general, the more recent samples from the point x estimate the statistics around x more

importantly, so the concept of kernel function is introduced. The formula 2||(||h

xxg i

shows that

the sample point ix affects the size of the current point x, that it has the right value. Therefore, mh(x)

is the kernel function g(x) and the weighted mean shift vector by ix .

Mean Shift vector always points to the direction of maximum density, which is guaranteed by the molecular term, while the denominator is reflected in each iteration step of moving the kernel function to a lesser density area. The step size is greater in the greater density area and the step size is smaller in the greater density area. That Mean Shift algorithm is a variable step size gradient rising algorithm, and it is called as the adaptive gradient rising algorithm. 2.2 Mean Shift algorithm is applied in this article

In this article the image segmentation of foreground and background for each image pixel, we select

a circle. Using the method mentioned calculates Mean Shift vector, and the center of this circular

window changes in accordance with )(1 xmhxx tt . Every time such an iterative algorithm will

eventually converge to a value. Therefore, each point is this operation for the image, and each point will converge to some value. If we only consider the selected colors of the kernel function, then the different pixels converges to the same value (or less), which shows that these points are belong to the same class.

As Mean Shift is a non-parametric clustering algorithm, so the time will not achieve the final parameter estimation results of performance caused by inappropriate variation. And Mean Shift has the advantages of fast calculation (Select a simple kernel function), so the algorithm can improve the overall performance.

Finally, we will compare Mean Shift segmentation algorithm with the watershed segmentation algorithm. Using two different image segmentation algorithms deals with the same picture, which compares their segmentation results. The results are shown in Figure 1.


128

(a) Original image (b) The marked image before segmentation

(c) Watershed segmentation (d) Mean Shift segmentation

Figure 1. The comparison of Mean Shift algorithm and watershed algorithm

By the comparison of Fig.1, we can draw the following conclusions: (1) Mean Shift is an image segmentation algorithm that it does not needs the parameters and the

initial conditions. The watershed algorithm needs given the initial conditions: Draw the area of the various objects, which is shown in Fig.1(b). The white part of the image is the mark before the segmentation.

(2) The segmentation results of Fig.1(c) and Fig.1(d) can be seen that Mean Shift is more reasonable than the watershed algorithm, and it can assign relatively good belonging to the same area, but the watershed algorithm is worse.

(3) In Mean Shift algorithm, the block of the same colors represents the same area, and the watershed algorithm is represented by the boundary line with a region. We can see by comparing that Mean Shift image segmentation algorithm is relatively complete and good for carrying out the following work, and the watershed segmentation results are relatively scattered.

2.3 The basic idea of the algorithm

In this section we will show how to separate the segmentation issue of foreground and background

into the maximum flow problem. In fact, this core idea is the use of graph theory to solve the problem of image separation. We can consider an image: foreground and background are all the complex by some of the color regional composition, so the separation of foreground and background is how to find such a segmentation to make the segmentation just the image border of the foreground and background. In the maximum posterior probability (MAP)--Markov random field (MRF) of the POTTS [46] model combining with the image segmentation method, we can separate the segmentation problem of the image of foreground into the minimization problem of the function. For an image, the greater the gap between the adjacent color regions, we define the weight function is smaller. Generally, there is a big


129

color difference in the border of the foreground and background, so the value of the weighting function in these regions is smaller than in other regions. In this case, the smallest segmentation of the entire image is most likely in the border of the foreground and the background. Therefore, we use the maximum flow to look for the minimum segmentations of the image and the separation of the foreground and the background of the image are naturally resolved.

In order to improve the performance of the algorithm, first, we use the Mean Shift algorithm to do the pre-segmentation of the image before we do the separation of the foreground and the background of the image, which will change the pixel-based processing into the region-based processing to improve the processing speed of the algorithm. Finally, some less than ideal segmentation parts will been processed by the digital matting techniques to obtain the desired segmentation results.

3. The steps of the algorithm

The whole processes of the segmentation of foreground and background are as follows: (1) Use Mean Shift algorithm to block the entire image, and to give each region a different color,

and the color is randomly given. (2) First, each region is given a number, and this number is from 1 to N. Where, N denotes the

image is the total number blocked. Then use the method to determine the image through various regions of the adjacent relationship. Finally, a relationship matrix records the results.

(3) The users cross in the image to determine which areas belong to the foreground and which areas belong to the background as the seed regions of the computer searching. In the following discussion, we use source calibrating the foreground area and use sink calibrating the background area.

(4) Using the color histogram estimates the color distribution of each region to determine the weights of the adjacent areas and the various regions to source and sink area. We know that the color histogram is an estimate of the color distribution in computer graphics and image processing, it is the count of some region of pixels to to obtain 2D or 3D color space. The weighting function is the final gotten value of two adjacent regions by comparing the color histogram of the Euclidean distance. Select A area and B area during the Mean Shift clustering, and we get a block diagram of an image shown in Fig.2.

(a)Original image (b) The partitioning effect of Mean Shift

Figure 2. The application of Mean Shift

In Fig.2, the same color in the middle of the area indicates that this pixel belongs to the same area. First, we built a counting frame of 3D by R, G, B, we divide each axis into 15 parts. Thus we get an RGB vector (15*15*15) of the region normalized. By calculating the Euclidean distance representing the space vector of the different regions, we will get the weight of w of two regions.

15

1

222 )()()[(1

1

ipikipikipiki BBPGRR

w (3)


130

k and p respectively are the k-th and p-th regions. And the calculation is relatively complex for the square and the radical. As , and finally selected the actual expression is the flowing formula:

15

1

|]||||[|1

1

ipikipikipiki BBGGRR

w (4)

The final results show the same effects are obtained by the two methods, but (4) is faster than (3).

We have clearly marked by the user that the foreground and the background respectively are

}{ FnK and }{ B

nK , and ||)(||min Fn

Fi KiCd is the minimum distance between the i

region and the marked foreground set. ||)(||min Bn

Bi KiCd is the minimum distance

between the i region and the marked background set, which, C(i) represents each region in the

RGB space vector, )1( Nixi is image area. If the region belongs to the foreground, we

get 1ix . If the region belongs to the background, we get 0ix . Now, we can get a weight

function by using the POTTS model of the Markov random field:

Pp Nqp

qpp LLELELE),(

21 ),()()( (5)

Where, pL represents a mark, 1E represents a value determined the nature (color) of the

region itself, and 2E indicates a color similar to the adjacent area. This weight function can be

expressed as:

If ix is marked as foreground:

)0(0)1( 11 ii xExE (6)

If ix is marked as background:

0)0()1( 11 ii xExE (7)

If ix is not marked:

22

11

||)()(||1

1||),(

)0()1(

jCiCxxxxE

dd

dxE

dd

dxE

jiji

Bi

Fi

Bi

iBi

Fi

Fi

i

(8)

(5) Using (2) determines the relationship between the matrix and (4) determining the weights, and we can form a connected graph from source to sink. In this picture we get that there are two marginal, namely n-links and t--links. N--links relationship is connected with the adjacent edge of the region, t-links are connected to source and sink in the different regions of the edge region [10]. By using max-flow algorithm [11], we can get the minimum segmentation of the images. The minimum segmentation of all the area is divided into two parts. After this division, these areas together with source are identified as foreground, and the areas connecting with sink are identified as background.

(6) After getting the rough segmentation image, the user must identify the problem areas, and then use Poisson Matting method to get the final processing results.

Poisson Matting includes two steps: (1) The input images are calculated approximately the gradient field. (2) After the user determines the field boundaries, Poisson equation of the gradient field is solved out to get the corresponding values. Detailed derivation is as follows:

We believe that any pixel of the image is formed by linear combination of foreground and background, namely:


131

FFI )1( (9)

The derivative of both sides of (7), we can get: BFBFI )1()( (10)

The process of solving is divided into two steps--global solution and local optimization. The main processing steps of global solution are: (1) The features based on the users defining the boundary of the image are divided into the

determined foreground area, the determined background area and the unknown area of calculating value. For some unknown point in the area, the previous domain area is the closest to the point of the pixel color value as its initial prospects. The background area to the point is the closest to the pixel color value as its initial background.

(2) Assuming foreground and background are relatively smooth, F and B are approximately equal to 0, corresponding the Poisson equation of (10):

0)1( BF (11)

Being simplified as:

IBF

1 (12)

(12) means and the gradient of the problem domain are proportional, and the gradient field can be determined directly on the original image.

Boundary conditions: For any point p of the image,

B

Fp p

p

,0

,1| is called Dirichlet

boundary conditions, which, F is the foreground area, B is the background area. By Gauss-

Seidel iteration or over-relaxation iteration [11] can solve (12). (3) value is greater than 0.95 points going to the foreground area, and value is less

than 0.05 points going to the background area. Repeat (2) and (3) until the entire areas calculated are classified into foreground and background areas, or iterations to get value little change so far. Partial solution of the main steps: Supposing in some local areas, the background changes more rapidly, F and B can not be approximate to 0, (10) can be written as:

)( DIA (13)

For such problems, first, do the global calculation, then the local users determine the borders, calculate value of the local area in accordance with a similar overall value. At this point the boundary is:

p

p

p

global

B

F

p

,

,0

,1

|

(14)

Which, F is the foreground area, B is the background area, is the unknown area.

Here is the steps of entire program implementation: (1) Input images. (2) Mean Shift algorithm is used the image segmentation, while the pixels belonging to the

same area are given the same color. (3) The segmented image based on the number of statistical areas for each region is set a

mark in accordance with the order. (4) The user crossed, which can determine that which belongs to the foreground and which

belongs to the background.


132

(5) The basis of the weight function is defined t-link and n-1ink weight and set up the source-sink map.

(6) Use the max-flow algorithm to calculate the source-sink map to get the minimum segmentation graph, and then use graph traversal. The foreground is connected with source, and sink is connected to the background.

(7) The users designate problem areas, namely they need to calculate the matte block. The Poisson Matting algorithm is used to compute value to improve Mean Shift determining a unnatural boundary so that we can get a relatively good results.

(8) Output foreground images. 4. Experimental results and analysis

The following pictures are obtained the results of the segmentation of foreground and background by the algorithm. We test the algorithm in three cases, namely: simple foreground and background, complex foreground and background with the different colors of the background area and the foreground area and complex foreground and background with the same colors of the background area and the foreground area. The crossed green area belongs to the background, and the crossed blue area belongs to the foreground. The first column is the original images, and the second column is the blocked images of Mean Shift algorithm, and the third column is the segmentation images. By comparison we can see that our algorithm performs well in the first two cases, but in the third case, in particular, the background is very similar to the foreground that the result is not very satisfactory. The main reason is too high similar to the foreground and the background so that the color histogram in the distance is closer. The maximum flow algorithm for segmentation is easy to divide into different objects in the same piece.

Case 1: Simple foreground and background:

(a) The original image (b) The image of Mean Shift (c)The segmentation image Figure 3. Simple foreground and background

Case 2: Complex foreground and background with the different colors of the background area and

the foreground area.


133

(a)The original image (b) the image of Mean Shift (c) The segmentation image

Figure 4. Complex foreground and background with the different colors

Case 3: Complex foreground and background with the almost same colors of the background area and the foreground area.

(a)The original image (b) The image of Mean Shift (c)The segmentation image

Figure 5. Complex foreground and background with the almost same colors 5. Conclusion

This paper proposes a new segmentation algorithm of foreground and background based on Mean Shift algorithm. This method makes use of Mean Shift theory, color histogram, cutout image segmentation method, the maximum flow algorithm and digital matting way, which is full use of their own advantage to result in a better segmentation results. The Mean Shift algorithm in the segmentation


134

of foreground and background will be pixel--based image processing into a region--based image processing to improve processing speed of this algorithm. Experimental results show that the proposed algorithm dealing with natural image effect is obvious, and you only need a small amount of intervention, and the boundaries of the image is natural, which can meet the users' needs of the multimedia special effects. 6. References [1] Blue Screening. Smith, A.R., Blinn, J.F, “Blue Screen Matting”, In SIGGRAPH96, Proceedings of

the 23rd annual conference on Computer graphics and interactive techniques, pp. 259-268, 1996. [2] Chi-Man Pun, Chan-Fong Wong, “Fast and Robust Color Feature Extraction for Content-based

Image Retrieval”, International Journal of Advancements in Computing Technology, vol. 3, no. 6, pp. 75-83, 2011.

[3] M. A. Ruzon, C. Tomasi, “Alpha estimation in natural images”, IEEE Conference on Computer Vision and Pattern Recognition, pp.18-25, 2000.

[4] EHillman, J. Hannah, D.Renshaw, “Alpha channel estimation in high resolution images and image sequences”, In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp.1063-1068, 2001.

[5] Y.Y.Chuang, B.Curless, D. Salesin, R.Szeliski, “A Bayesian approach to digital matting”, In Proceedings of IEEE Computer Vision and Pattern Recognition, pp.264-271, 2001.

[6] Y. Y.Chuang, A. Agarwala, B. Curless, D．Salesin, R. Szeliski, “Video Matting of Complex Scenes”, ACM Transactions on Graphics, vol. 21, no. 3, pp. 243-248, 2002.

[7] Shengyou Lin, Jiaoying Shi, “Perception Based Alpha Estimation in Natural Image Matting”, In Proceedings of IEEE International Conference on Multimedia and Expo, pp. 1407-1410, 2004.

[8] Perez. P, GANGNET. M, BLAKE.A, “Poisson Image Editing”, ACM Transactions on Graphics, vol. 22, no. 2, pp. 313-318, 2003.

[9] Chi-Man Pun, Ning-Yu An, “Image Segmentation using Effective Region Merging Strategy”, JDCTA: International Journal of Digital Content Technology and its Applications, vol. 5, no. 8, pp. 59- 69, 2011.

[10] Ling Zhang, Yuan Zhang, “A Novel Region Merge Algorithm based on Neutrosophic Logic”, JDCTA: International Journal of Digital Content Technology and its Applications, vol. 5, no. 7, pp. 381-387, 2011.

[11] D Greig, B Porteous, A Seheult, “Exact Maximum a Posteriori Estimation for Binary Images”, Journal of the Royal Statistical Society, vol. 2, no. 51, pp. 271-279, 1989.


135

Documents

Color Image Segmentation of Foreground and Background ......understand the object, it is necessary to segment it from its image. The segmentation of foreground and background also