Level Set with Embedded Conditional Random Fields and Shape Priors for Segmentation of Overlapping Objects

8/9/2019 Level Set with Embedded Conditional Random Fields and Shape Priors for Segmentation of Overlapping Objects

1/12

Level Set with Embedded Conditional Random

Fields and Shape Priors for Segmentation of

Overlapping Objects

Xuqing Wu and Shishir K. Shah

Department of Computer Science, University of Houston, Houston, TX 77204, USA

Abstract. Traditional methods for segmenting touching or overlappingobjects may lead to the loss of accurate shape information which is akey descriptor in many image analysis applications. While experimen-tal results have shown the effectiveness of using statistical shape priorsto overcome such difficulties in a level set based variational framework,problems in estimation of parameters that balance evolution forces fromimage information and shape priors remain unsolved. In this paper, weextend the work of embedded Conditional Random Fields (CRF) by in-corporating shape priors so that accurate estimation of those parameterscan be obtained by the supervised training of the discrete CRF. In addi-tion, non-parametric kernel density estimation with adaptive window sizeis applied as a statistical measure that locally approximates the variationof intensities to address intensity inhomogeneities. The model is tested

for the problem of segmenting overlapping nuclei in cytological images.

1 Introduction

Segmentation of multiple objects in images wherein the objects may overlap orocclude other ob jects/regions of interest remains a challenging problem. This iscertainly the case for analysis of cytological images, wherein the classificationof the imaged sample is based on the shape, distribution and arrangement ofcells/nuclei, and other contextual information [1]. To automate this classifica-tion process, the computer needs to learn cellular characteristics by extractingcorresponding image features from the region of interest (ROI). The accuracy of

feature extraction depends on the performance of segmentation results. However,reliable segmentation of cytological images remains challenging because of inten-sity inhomogeneity and the existence of touching and overlapping cells/nuclei. Liet al. [2] gave a brief review on some methodologies towards the development ofautomated methods for nuclei or cell segmentation. Among all proposed meth-ods, thresholding, watershed, and deformable models or active contours remainthe most popular approaches. In general, watershed segmentation is claimed tobe more effective in terms of differentiating connected nuclei or cells. However,the watershed model for segmentation of multiple objects is not suitable forseparating overlapping objects because any segmented pixel/set is generally as-sociated to a single label and a pixel cannot be granted membership to more


2/12

2 Xuqing Wu and Shishir K. Shah

than one label. For example, fig.1.a shows two objects A and B with an overlap-ping areaC. Fig.1.b shows the expected segmentation result under the watershedscheme and fig.1.c presents the ground truth with two elliptical objects indicatedby red and blue contours. Accurate description of the shape is lost in the seg-

a b c

Fig. 1.Overlapping ellipse example: there are two objects in this image, A and B.

mented image shown in fig.1.b. Coupled curve evolution [3] and multiphase levelset [4] provide a powerful framework for segmenting multiple connected objects.In addition, shape priors can also be easily integrated into the level set formula-tion resulting in a more constrained segmentation [5]. Nonetheless, these modelsare subject to the same labeling constraints so as to avoid assigning differentlabels to overlapping regions. A practical solution to overcome this limitationis to evolve each curve independently [6] and let the shape prior control theboundary in the overlapping region.

One of the major drawbacks of the aforementioned continuous level setmethod is the ad-hoc style of the parameter estimation [7]. In order to leveragethe power of structured discriminative models like Conditional Random Fields(CRF), Cobzas and Schmidt [7] proposed to embed CRF into the level set for-mulation to improve edge regularization, incorporate correlations in labels ofneighboring sites and evaluate weighting parameters for each energy term bypre-trained discrete CRF models. A key problem that remains unsolved is theexplicit integration of shape information in a discrete CRF model so that pa-rameters that balance the influence from image information and shape priorscan be evaluated systematically through embedding. Our work addresses thisvery problem and is distinguished by three key contributions. First, we proposea novel spatial adaptive kernel density estimator to associate local observationswith expected labels. Second, we apply the top-down CRF model [8] as a con-straint and embed it into the level set formulation as the shape measurement.Third, we solve the parameter estimation problem for the curve evolution withshape priors by embedding potential functions and parameters learned from thediscrete CRF model.

The rest of the paper is organized as follows. Section 2 briefly introduces theCRF embedding method proposed by Cobzas and Schmidt [7]. The proposedadaptive kernel density estimator and embedded shape prior are presented insection 3. Estimation of parameters and the experimental evaluation of the de-veloped model for the problem of segmenting overlapping nuclei in cytologicalimages are presented in section 4. The paper is concluded in section 5.


3/12

Segmentation of Overlapping Objects 3

2 Level Set Methods with Embedded CRF

Image segmentation is a fundamental problem in image analysis and can beviewed as classification or labeling of image pixels/regions, wherein y representsa set of labels to be assigned to each pixel represented by its value x. Sincemost natural images are not random collection of pixels [9], it is importantto find an accurate model to describe information in the image to achieve goodsegmentation accuracy. Both CRF and level set are popular mathematical modelsused for solving segmentation problems.

A binary CRF with weighted Ising potentials for the task of segmentation is

formulated as follows [7]:

p(y|x,w,v) = 1Z

exp

iN

yiwTfi(x) +

i,jE

(1 |yi yj |)

k

vkfijk (x)

,

(1)where Z is a normalizing constant known as the partition function, N is thenumber of nodes, fi(x) denotes features for node i, and fijk represents the k

th

feature for the edge between nodes i and j. Node labels are yi. The associationtermyiw

Tfi(x), denoted asAifrom here on, relates the observationxto the labelyi. This term corresponds to the level set data energy (Eq.2) that is inside oroutside of the contour. The interaction term (1 |yi yj |)

kvkfijk (x), denoted

byIij from here on, is a measure of how the labels at neighboring sites should

interact. This term is analogous to the regularization energy (Eq.2) in the levelset model. In particular, fijk (x) is defined as

11+|fik(x)fjk(x)|

. Because Eq.1 is

jointly log-concave in w and v, we can find the global maximum in terms ofwand v.

Level set methods define the evolving curve C in R2. C is implicitlyrepresented as the zero-level of an embedding signed distance function : R. In particular, to segment an image into two disjoint regions 1 and 2, theenergy functional of a continuous model with embedded CRF has the followingform:

E() =

H()(wTf) + (1 H())(wTf)

data term+ |H()|

k

vk1

1 + |fk|

regularization termdx ,

(2)where k is the index for each edge feature and vk 0. The associated Euler-Lagrange equation is:

t =()

2wTf+ div

k

vk1

1 + fk

||

, (3)

where is the regularized version of the delta function . The new continuousformulation takes the advantage of the embedded CRF model whose parameters{w, v}can be estimated by the supervised training in its discrete counterpart.


4/12


3 Adaptive Kernel Density Estimation and Embedded

Shape Priors

In general, the discriminative model wTf works well when the test image canbe well predicated through the supervised regressive training on sample images.However, when the variance of the difference between predicated and observedvalues is not well contained, the resulting segmentation can be ill-conditioned.This is especially the case for segmenting cells/nuclei from cytological smears be-cause of the intensity inhomogeneity caused by the cell chromatin, the existenceof dense intracellular as well as intercellular matter. In this paper, we propose touse a non-parametric density estimation approach to capture the local variationwithout imposing restrictive assumptions. In addition, we apply the top-downCRF model [8] for the shape constraint and derive its corresponding embeddedterm in the continuous space. Thus the weighting factor of shape priors can beestimated from the discrete CRF model.

3.1 Adaptive Kernel Density Estimation

Parzen window is a classical non-parametric kernel density estimation method.Nonetheless, the traditional approach of using a fixed window size h imposesnegative effects on segmentation results when there are geometrically disjointregions in the image, because not every data point makes an equal contributionto the final density estimation of a pixel i in the 2D space. For example, pixels

located far away from pixel i should weigh less in the formulation. To have thebandwidth change with the spatial distance, we define the adaptive window sizehij as:

hij =

1

N

14 1

exp

d2ij

2

, (4)

wheredij is the Euclidean distance between pixel i and j in the 2D space, is auser defined scale parameter, and Nis the size of sample points. Thus, the newestimated intensity density function at pixel i is:

p(xi) = 1

N

Nl=1

1

hilKxi xlhil

, (5)

whereK

() is the Gaussian kernel, and xl is a sample drawn from population

with density p(xi). The accuracy of the estimated density is increased by usingsmall bandwidth in regions that are near the pixel i since samples in the nearneighborhood provide better estimation. On the other hand, a larger bandwidthis more appropriate for samples located far away. To be less sensitive to noise,we could convolve image x with a Gaussian kernel so that samples xl in Eq.5are replaced by a smoother version xl. Discrete CRF model for segmenting animage into two disjoint regions 1 and 2 can now be re-written as:

p(y|x,w,v) = 1Z

exp

iN

wklogp(fi(x)|k) +

i,jE

Iij

k {1, 2} , (6)


5/12


wherew logp(fi(x)|k) forms the association term Ai, Iij is the same as inEq.1, and p(fi(x)|k) is estimated using the adaptive kernel as:

p(fi(x)|k) = 1Nk

Nkl=1

1

hilKfi(x) fl(x)

hil

l k , (7)

where fl(x) is a smoother version of the node feature fl(x). Embedding CRFmodel of Eq.6 into the functional in the continuous space, the new contourenergy,Ecv, in the level set counterpart takes on the form:

Ecv() = H()w1logp(f(x)|1) (1 H())w2logp(f(x)|2)+|H()|

k

vk1

1 + |fk|dx , (8)

whose parameters {w, v} are trained by the discrete CRF. Minimizing the energyin Eq.8 is accomplished by solving the following Euler-Lagrange equation:

t =()

wT

logp(f(x)|1) logp(f(x)|2)

+ div

k

vk1

1 + fk

||

.

(9)Fig.2 shows that, compared to the local association model with Gaussian regionstatistics, the non-parametric approach introduced in Eq.9 is less sensitive to thevariation of intensity values. This results in improved segmentation even withoutthe introduction of shape priors for non-occluding objects. We note that therehas been a previous proposal, called local binary fitting energy (LBF) model, forfitting energy based intensity information in local regions [10]. Compared to theLBF model, the kernel function Kused in Eq.4 is a well-behaved kernel [11] andthe asymptotic convergence of the kernel density estimator is guaranteed [12].Unlike the LBF model, parameters of Eq.8 are estimated systematically throughthe discrete CRF model.

010

2030

4050

6070

8090

100

0

20

40

60

80

100

80

85

90

95

100

105

110

115

120

(a) (b) (c)

Fig. 2. Comparative results for segmenting images with intensity inhomogeneity (a)Segmentation result using adaptive kernel density estimation. (b) Segmentation resultusing Gaussian model. (c) Intensity profile of the area inside of the nucleus


6/12


3.2 Embedded Shape Priors

The occluded boundary of overlapping regions cannot be detected by algorithmsthat only use edge or region information. Statistical shape models have beendeveloped to overcome these difficulties [5]. Inspired by the work of Levin etal. [8] and Cremers et al. [12], we propose an explicit integration of shape priorsinto the discrete CRF and derive embedding entities in the level set model.Therefore, parameters that weigh the influence from shape information can beestimated accurately by training the CRF model.

Let an implicit representation of a shape be a binary image patch with la-beling of

{0, 1

}. Let c be a binary image patch representing a shape, c(i)

be the label of pixel i. The distance between two shapes a and b is definedas d2(ca,

cb) =

Ni=1(

ca(i) cb(i))2, where N is the size of the image patch.

Given a set of training shapes ci :i = 1...M, the probability density of a shapec can be estimated as:

p(c) = 1

M

Mi=1

exp

1

22d2(c, ci )

, (10)

where 2 can be estimated by 1M

Mi=1minj=i d

2(ci , cj ) [12]. Let Su0 = z logp(cu0)

be the potential associated with the image patch c centered at u0 with binarylabel y andp(cu0) is the probability density of a shape

c centered atu0. Incor-porating this into the discrete CRF model results in:

p(y|x,w,v,z) = 1Z

exp

iN

Ai+

i,jE

Iij + Su0

. (11)

Ai and Iij are defined as in Eq.6. Parameters{w,v,z} are estimated throughthe training of the discrete CRF model.

To embed the shape prior into the level set, we re-define the implicit shaperepresentation as the signed distance function. The distance between twoshapes a and b in the level set formulation can be given by:

d2L(a, b) = (H(a(x)) H(b(x)))2dx , (12)

where H() denotes the Heaviside function. The density of a shape in the levelset formulation, represented by pL, is given as:

pL(u0) = 1

M

Mi=1

exp

1

22d2L((x+ u0), i(x))

, (13)

where u0 accounts for translation in continuous space. Let the shape energyEshape =z log(pL(u0)), then the total energy of the contour in the level setformulation is the sum ofEcv defined in Eq.8 and Eshape, whose parameters


7/12


{w,v,z}are estimated by the discrete CRF. Total energy is then minimized by:

t =()

wT

logp(f(x)|1) logp(f(x)|2)

+ div

k

vk1

1 + fk

||

+

z

exp( 1

22Ld2L((x+ u0), i(x)))(H((x+ u0)) H(i(x)))2L

exp( 1

22Ld2L((x+ u0), i(x)))

, (14)

where 2L is evaluated by 1M

M

i=1minj=i d2L(i, j ).

3.3 Construction of Shape Priors

With the objective of segmenting nuclei, a set of contours of elliptical shapepriors was generated according to the general parametric form of the ellipse:

x(t) = a cos t cos b sin t sin y(t) = a cos t sin + b sin t cos ,

(15)

where t [0, 2]. is the angle between the X-axis and the major axis of theellipse. The ratio b

ais a measure of how much the conic section deviates from be-

ing circular. In this paper, we construct shape prior with ba {

0.4, 0.6, 0.8, 1.0}

.The angle of the major axis, , was also varied from 0 to in steps of

8. Fig.3

shows the array of elliptical training shapes. Each training shape is representedby a 32 32 patch. Rotation invariance is integrated into the estimation of thedensity of a shape (Eq.10 or Eq.13) by using this set of training shapes.

Fig. 3. Training shapes for elliptical objects with different eccentricities and rotationangles


8/12


4 Implementation and Experimental Results

Images used for training and testing are obtained from slides of cytologicalsmears that are Papanicolaou or Diff-Quik stained for the purpose of differ-entiating pathologies in fine needle aspirated cells from thyroid nodules. Imagesare acquired by using an optical microscope and a grayscale CCD camera having768 512 pixels (9 9m) at 8-bit digitization. Manual segmentation was per-formed to establish the ground truth. The segmentation accuracy is measuredby global consistency error, a well-established standard measurement used inimage segmentation problem [13].

To evolve the curve according to eq.14, an iteration scheme is implementedby discretizing the PDE [4]. The level set function is initialized as a step func-tion that takes negative values inside the contour and positive values outside ofthe contour. The initial contour is obtained from the segmentation result usingwatershed and unsupervised clustering [14]. Since the watershed lines separatethe image into non-overlapped regions, we can evolve each closed watershed lineof a region independently.

Four parameters are estimated by training the discrete CRF with shape pri-ors [15]:w1 is related to image information inside of the contour; w2is for imageinformation outside of the contour; v governs the force defined by edge features;z balances the influences from the shape prior. We have selected 20 images forthe training: ten images with Papanicolaou stain and ten images with Diff-Quikstain. Each of the image contains at least one pair of overlapping nuclei. Pa-

rameters are trained separately for different stains. Table.1 lists the value ofparameters trained for different stained images. Fig.4 shows that if the processof curve evolution is dominated by edge features, the evolved contour can notarrive at the boundary of the nucleus of interest. Although a global optimal selec-tion of parameters in terms of segmentation accuracy can not be guaranteed [15],parameters estimated by the proposed scheme can be treated as local optimalselections. To further evaluate this local optimal property of selected parametersobtained through the CRF training, we added a small perturbation to the valueof these parameters by varying the ratios w1

w2and z

v. Fig.5.a and fig.5.b show the

segmentation accuracy across the entire training set under varying parameters.It can be seen that the selected parameters from CRF training have the bestsegmentation performance.

Table 1. Parameters Estimation Results From Training

Stain w1 w2 v zPapanicolaou 1.0 1.0 3.25e3 8e5

Diff-Quik 1.0 2.0 1.73e3 9e5

To quantify the performance of the proposed method, a total of 40 testingimages were selected. Half of these images were Papanicolaou stained while the


9/12


(a) (b)

Fig. 4.Comparative results for parameters estimation. (a) Contour of balanced forces;(b) Contour of weak constraints from shape prior

other half are Diff-Quik stained. The performance of the proposed model isevaluated with two criteria, the accuracy of the segmentation in terms of thearea occupied by nuclei and the number of nuclei correctly counted by eachsegmented region.

First, we evaluated potential benefit of using non-parametric adaptive densityestimation to model variations of intensities in cytological images. We found thatthe average segmentation accuracy using the proposed model integrated withadaptive density estimation is 81%, as opposed to 72% accuracy rate obtained

by the using the Gaussian regional model integrated in the overall framework.

Second, we compared our method with two other popular models, water-shed [14] and CRF [16]. Test results listed in table.2 show that the proposedmodel has the best overall performance in terms of segmentation and nucleicounting accuracy. The low accuracy of nuclei counting by using CRF modelis expected due to its tendency to merge adjoining regions together as shownin fig.6.c and fig.6.f. Compared to the CRF model, although watershed is moreeffective in differentiating overlapping nuclei, it has two drawbacks. First, theshape information of nuclei, a key morphological feature in cytological classi-fication, could be distorted because of the unique labeling constraint. This isattributed to the fact that each pixel in the overlapping region could have two

different labels when two nuclei overlap. Second, the model used for constructingwatershed lines is dominated by the image intensity only, which is very sensitiveto the intensity inhomogeneity. As a result, its segmentation accuracy in termsof the area occupied by the nuclei is lower than CRF, which models both lo-cal and neighborhood information. The proposed method inherits the modelingadvantage of CRF and can model non-local dependencies during the curve evolu-tion [7]. Because each contour evolves independently, pixels in overlapping areascan be assigned multiple labels as indicated in fig.6.a and fig.6.d. Fig.7 presentsanother example of segmentation results after applying the proposed method,watershed and CRF to Papanicolaou and Diff-Quik stained images, respectively.


10/12


(a)

(b)

Fig. 5. Comparative segmentation results for parameters estimation. (a) Varying theratio w1

w2(b) Varying the ratio z

v

5 Conclusion

Shape information is a key descriptor in analyzing cytological images. To recoverthe shape information for overlapping objects after segmentation, we extend theembedded CRF model by incorporating shape priors. In contrast to other ap-proaches, parameters that govern the force defined by the image informationand shape priors are estimated accurately through the training of discrete CRF.By evolving each contour independently, original shape information is recov-ered by assigning multiple labels to the common region shared by overlappingnuclei. In addition, we have presented an adaptive scheme for non-parametricdensity estimation based on intensity information in local regions. The proposedmethod is tested for the problem of segmenting overlapping nuclei with intensityinhomogeneity while preserving shape information of occluded objects.


11/12


(a) Proposed method (b) Watershed (c) CRF

(d) Proposed method (e) Watershed (f) CRF

Fig. 6. Sample segmentation results using different models. (a)(b)(c) Papanicolaoustained sample. (d)(e)(f) Diff-Quik stained sample

Table 2. Comparison of accuracy of segmentation and nuclei counting

Model Accuracy of segmentation Accuracy of nuclei countingProposed method 81% 94%

CRF 79% 52%

Watershed 74% 94%

References

1. Amrikachi, M., Ramzy, I., Rubenfeld, S., Wheeler, T.: Accuracy of fine-needleaspiration of thyroid. Arch Pathol Lab Med 125 (2001) 484488

2. Li, G., Liu, T., Nie, J., Guo, L., Chen, J., Zhu, J., Xia, W., Mara, A., Holley,S., Wong, S.: Segmentation of touching cell nuclei using gradient flow tracking.Journal of Microscopy 231 (2008) 4758

3. Fan, X., Bazin, P.L., Prince, J.L.: A multi-compartment segmentation frameworkwith homeomorphic level sets. In: CVPR. (2008)

4. Vese, L., Chan, T.: A multiphase level set framework for image segmentation usingthe mumford and shah model. IJCV 50 (2002) 2712935. Rousson, M., Paragios, N.: Shape priors for level set representations. In: ECCV.

(2002)6. Mosaliganti, K., Gelas, A., Gouaillard, A., Noche, R.: Detection of spatially corre-

lated objects in 3d images using appearance models and coupled active contours.In: MICCAI. (2009)

7. Cobzas, D., Schmidt, M.: Increased discrimination in level set methods with em-bedded conditional random fields. In: CVPR. (2009)

8. Levin, A., Weiss, Y.: Learning to combine bottom-up and top-down segmentation.IJCV 81 (2009) 105118

9. Kumar, S., Hebert, M.: Discriminative random fields. IJCV68 (2006) 179201


12/12


(a) Proposed method (b) Watershed (c) CRF

(d) Proposed method (e) Watershed (f) CRF

Fig. 7. Segmentation results using different models. (a)(b)(c) Papanicolaou stainedsample. (d)(e)(f) Diff-Quik stained sample

10. Li, C., Kao, C.Y., Gore, J.C., Ding, Z.: Implicit active contours driven by localbinary fitting energy. In: CVPR. (2007)

11. Mittal, A., Paragios, N.: Motion-based background substraction using adaptivekernel density estimation. In: CVPR. (2004)

12. Cremers, D., Osher, S.J., Soatto, S.: Kernel density estimation and intrinsic align-ment for shape priors in level set segmentation. IJCV 69 (2006) 335351

13. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented naturalimages and its application to evaluating segmentation algorithms and measuring

ecological statistics. In: ICCV. (2001)14. Wu, X., Shah, S.: Comparative analysis of cell segmentation using absorption and

color images in fine needle aspiration cytology. In: IEEE International Conferenceon Systems, Man and Cybernetics. (2008)

15. Szummer, M., Kohli, P., Hoiem, D.: Learning CRFs using graph cuts. In: ECCV.(2008) 582595

16. Wu, X., Shah, S.: A bottom-up and top-down model for cell segmentation usingmultispectral data. In: ISBI. (2010)

Documents

Level Set with Embedded Conditional Random Fields and Shape Priors for Segmentation of Overlapping Objects