A semisupervised segmentation model for collections of images.bak

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 6, JUNE 2012 2955

A Semisupervised Segmentation Model forCollections of Images

Yan Nei Law, Hwee Kuan Lee, Michael K. Ng, and Andy M. Yip

Abstract—In this paper, we consider the problem of segmenta-tion of large collections of images. We propose a semisupervisedoptimization model that determines an efficient segmentation ofmany input images. The advantages of the model are twofold.First, the segmentation is highly controllable by the user so thatthe user can easily specify what he/she wants. This is done byallowing the user to provide, either offline or interactively, some(fully or partially) labeled pixels in images as strong priors for themodel. Second, the model requires only minimal tuning of modelparameters during the initial stage. Once initial tuning is done,the setup can be used to automatically segment a large collectionof images that are distinct but share similar features. We willshow the mathematical properties of the model such as existenceand uniqueness of solution and establish a maximum/minimumprinciple for the solution of the model. Extensive experimentson various collections of biological images suggest that the pro-posed model is effective for segmentation and is computationallyefficient.

Index Terms—Biological image segmentation, image segmenta-tion, interactive, microscopy images, multiple images.

I. INTRODUCTION

I MAGE segmentation is used in many areas, including com-puter vision, computer graphics, and medical imaging, to

name a few. Fully automatic image segmentation has many in-trinsic difficulties and is still a very hard problem. For example,it is very often that an image can have many segmentationsthat are meaningful. Therefore, the segmentation problem is illposed if no additional knowledge about the desired segmen-tation is given. However, in many applications, such as cellsegmentation in microscopy images and organ segmentation inmedical images, the kind of objects and segmentation of interestare known in advance. It is therefore tempting to design seg-mentation methods that allow the user to specify what he/she

Manuscript received July 26, 2011; revised December 15, 2011; accepted Jan-uary 23, 2012. Date of publication February 13, 2012; date of current versionMay 11, 2012. The works of Y. N. Law and H. K. Lee were supported in partby the Biomedical Research Council of A STAR, Singapore. The work of M.K. Ng was supported in part by the Hong Kong Research Grants Council underGrant 201508 and HKBU FRGs. The work of A. M. Yip was supported in partby the Academic Research Grant R-146-000-139-112 from the National Uni-versity of Singapore, Singapore. The associate editor coordinating the reviewof this paper and approving it for publication was Prof. Xiaolin Wu.Y. N. Law and H. K. Lee are with the Bioinformatics Institute, Singapore

138671 (e-mail: [email protected]; leehk @bii.a-star.edu.sg).M. K. Ng is with the Department of Mathematics, Hong Kong Baptist Uni-

versity, Kowloon, Hong Kong (e-mail: [email protected]).A. M. Yip is with the Department of Mathematics, National University of

Singapore, Singapore 119077 (e-mail: [email protected]).Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TIP.2012.2187670

wants. In semiautomatic and interactive image segmentation,the user marks some sample pixels from each class of objects.The computational algorithm then computes a classification ofother pixels. This way, the resulting segmentation is highly con-trollable by the user and thereby eliminates much ambiguity indefining a partition. Because of their desirable properties, therehave been increasing activities in the research community to de-velop interactive semiautomatic image segmentation algorithms[1]–[11].In particular, Guan and Qiu [2] has recently developed an op-

timization-based two-class segmentationmodel, in which an op-timal class membership function is computed through the min-imization of a quadratic cost function with user-supplied sam-ples as linear constraints. The basic idea is that two pixels shouldhave similar membership if they are either geometrically sim-ilar or photometrically similar or both. The results are quite im-pressive. The model was later extended in [12] to handle themultiple-class problem. Some effective numerical optimizationmethods and fundamental theoretical properties of the modelswere studied in [13]. Some related models for generic data clas-sification include [14] and [15]. In [16], the aforementionedsingle-image optimization models were extended to the mul-tiple-image case in the context of image retrieval. However,the method assumes the availability of a suitable dictionary thattypically contains thousands of texture patches (prototypes) toserve as a common ground for comparison between images.While these models have been very successful in segmenting

single images, it is clear that they will have an even strongerimpact if they can be used for segmentation for multiple im-ages, which is fundamental to many applications that use largecollections of similar images. The idea here is that the user canspecify some sample pixels or objects in one or more imagesonce and for all so that the segmentation of all other images isfully automatic. This way, the overhead due to the initial manualintervention is totally justified.In this paper, we consider the problem of segmentation of

large collections of images. We propose a semisupervised op-timization model that determines an efficient segmentation ofmany input images. The advantages of the model are twofold.First, the segmentation is highly controllable by the user so thatthe user can easily specify what he/she wants. This is doneby allowing the user to provide, either offline or interactively,some labeled pixels in images as strong priors for the model.Second, the model requires only minimal tuning of model pa-rameters during the initial stage. Once initial tuning is done, thesetup can be used to automatically segment a large collectionof images that are distinct but share similar features. We willshow themathematical properties of themodel such as existence

1057-7149/$31.00 © 2012 IEEE

http://ieeexploreprojects.blogspot.com

2956 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 6, JUNE 2012

and uniqueness of solution and establish a maximum/minimumprinciple for the solution of the model. Extensive experimentson various collections of biological images suggest that the pro-posed model is effective for segmentation and is computation-ally efficient.The outline of this paper is as follows. In Section II, we

present the proposed model and discuss the properties and thecomplexity of the model. In Section III, we show the experi-mental results of the proposed model. In Section IV, we studythe effect of various model parameters. In Section V, some con-cluding remarks are given.

II. IMAGE SEGMENTATION MODEL

In this section, we present the formulation of the proposedmodel. For ease of presentation, we illustrate the two-imagemultiple-class case. This two-image model can be used to seg-ment a collection of images one at a time. The generalization tomultiple-image multiple-class case is clear.

A. Optimization Model

Let for be two given multichannel images. Theirsizes are not necessarily the same. Let be the set of all pixelsin image . Let be the set of all unlabeled pixels in image. Let be the set of pixels in image labeled to one of

the classes by the user. Thus, . Here, weallow both images to contain labeled and unlabeled pixels forthe sake of generality. The set of labeled pixels is divided into

, where is the set of pixels that are labeledwith class , for .Let if , and let if , so that is an

index referring to an image different from the image indexed by. For each pixel and each pixel , let be asimilarity score between the pair of pixels, for , . When

, the similarity is computed within image ; when, the similarity is computed across two images. For each, it is assumed that the similarity scores are normalized

such that

(1)

The precise definition of the similarity score used in this paperis given in the next subsection.For each pixel , let be a set of pixels in

image , which is called the neighbor of in . The neighbor isprecisely defined in (11) and (12). Presumably, these pixels havehigh similarity scores with . For each , letbe the degree of membership of pixel to class . Itis required that . We also denote by thevector .The basic idea of the model is that the memberships of sim-

ilar pixels should be similar. For each unlabeled pixel ,the membership to class inferred from its neighbors is theweighted average, i.e.,

To ensure that such an inferred value is close to for each, , and , a least-squares fitting approach is used. It amountsto solving the following quadratic minimization problem:

(2)

subject to the (componentwise) constraints

(3)

(4)

for and and the “boundary conditions”

forfor

(5)

for and .To simplify the notation, let for

and , where for .Let be the down-sampling matrix from to . Let

(6)

(7)

(8)

(9)

The objective function (2) can be compactly written in matrixform as

(10)

where is the Euclidean norm.

B. Similarity Measures

Two kinds of similarity measures, namely, geometric andphotometric, are considered. The former is based on pixellocations, whereas the latter is based on color features.For each pixel , its geometric neighbor is

defined as

where is a constant controlling the size of the window,and is the vector maximum norm. We often set sothat a 3 3 window around pixel is used. Note thatand the geometric neighbor is not defined across two images.The geometric similarity is defined as

ifotherwise

where is a normalization constant such that ,and is computed as the sample variance of the geometriclocations within .


LAW et al.: SEMISUPERVISED SEGMENTATION MODEL FOR COLLECTIONS OF IMAGES 2957

For each pixel , let be its feature vector. In our ex-periments, we use the RGB values over a 3 3 window aroundpixel to construct a feature vector of dimension 27. Then, thewithin-image photometric neighbor is defined to bethe top 4 pixels within the 17 17 window around pixel (ex-cluding pixel itself), whose feature vectors are nearest to (inEuclidean norm). Using a larger window size allows us to con-nect photometrically similar pixels that are further apart. How-ever, doing so will increase the computational cost. The choiceof the size 17 17 is a balance between both extremes. Thewithin-image photometric similarity is defined as

ifotherwise

where is a normalization constant such that ,and is computed as the sample variance of the photometricfeatures within .Consider as an unlabeled image and as a (fully or

partly) labeled image so that is nonempty. Let . Tomake the across-image photometric neighbor effi-cient to compute, we define it to be the top 4 labeled pixels insome , whose feature vectors are nearest to . Here,is a random sample of such that it contains an equal numberof pixels from and . The across-image photometricsimilarity is defined as

ifotherwise

where is a normalization constant such that ,and is computed as the sample variance of the photometricfeatures within .The within-image neighbor and the across-image neighbor

are defined, respectively, by

(11)

(12)

for . The combined within-image similarity and com-bined across-image similarity are

(13)

(14)

respectively, where is a tuning parameter controllingthe weight between geometric and photometric similarities, and

is a tuning parameter controlling the weight betweenwithin- and across-image similarities. In matrix form, we have

for . Each row sum of the concatenated matrixis 1.

Fig. 1. Original image and its ground truth segmentation boundaries (greenlines).

Fig. 2. Breast cancer cell images. Box plots of (a) the accuracy of the segmen-tations and (b) the -measure for the nucleus region retrieved by the proposedmultiple-image model (MI) with and , SVM, KNN, KMeans,and SSKMeans.

The proposed model has only parameters and to tune.When applied to the segmentation of a collection of images, theyonly need to be tuned once. In Section II-D, we describe howthey are tuned in practice.

C. Optimality Conditions

By differentiating the objective function in (10) and intro-ducing Lagrange multipliers for the constraints (3)–(5), it canbe shown that the optimality conditions are given by the linearsystems (see Theorem 1 in the Appendix), i.e.,

for , where

(15)

(16)

Here, denotes matrix transpose, and denotes an identitymatrix. The components of are given by

ififif .

(17)

The solution satisfies the weak discrete maximum/minimumprinciple, which guarantees the satisfaction of the constraints

, provided that and that each



Fig. 3. Segmentation of two of the testing images obtained by the proposedmethod (MI) and others (the green lines depict the segments’ boundaries). Theground truth segmentation in Fig. 1 is used as the training set for the MI, SVM,KNN, and SSKMeans methods.

unlabeled pixel can be connected to a labeled pixel througha sequence of directed edges, each of which connects a pixelto one of its neighbors in the same image or a different image

Fig. 4. Original image and its ground truth segmentation.

Fig. 5. Retinal images. Box plots of (a) the accuracy of the segmentations and(b) the -measure for the blood vessels retrieved by the proposed multiple-image model (MI) with , SVM, KNN, KMeans, and SSKMeans.

(see Theorems 2 and 3 in the Appendix). It follows that isnonsingular and that the solution is unique (see Theorem 4 inthe Appendix). When the matrix size is not large (e.g., ),these linear systems can be often efficiently solved by Gaussianelimination using standard machines. However, for a largerimage size, preconditioned iterative methods are recommended(see, for instance, [13]).

D. Application to a Collection of Images

Suppose that contains some manually labeled pixels whileother images are unlabeled. To segment a collection of images,we apply the multiple-image model (2)–(5) to and one otherimage (called ) at a time. Thus, we have and .A simple way to apply the model is to let for all

, so that , and let .In this case, the matrix is block upper triangular, i.e.,

is then decoupled from and is solely determined frominformation within itself. Once is determined, canbe computed by solving the linear system

Moreover, has to be computed only once for segmentingthe entire collection.In our experiments, the parameters and are manually

tuned when we solve for and the first unlabeled image.These values are then used to segment all other images. This



Fig. 6. Segmentation of five of the testing images obtained by the proposed method (MI) and others. The ground truth segmentation in Fig. 4 is used as the trainingset for the MI, SVM, KNN, and SSKMeans methods.

method has worked very well, and the tuning is quite easy forthe collections that we used.If there are images that contain labeled pixels, then we

can simply apply the model for images to segment oneunlabeled image at a time. The model we proposed is easy to beextended.We remark that, for the single-image case, we can show thatsatisfies the strong maximum principle, which guarantees

the strict inequalities and the uniqueness of

[13]. For the multiple-image case, if , then we onlyhave the weak maximum principle, which impliesonly. However, themore important uniqueness of still holds.(See Theorem 3 in the Appendix for details.)

E. Computational Complexity

Regarding the multiple-image model in Section II-D, we dis-cuss its major computational costs (in terms of flop counts) ineach of the following steps.



Step 1) Compute (independent of and ).Step 2) Compute (dependent on ) and solve the linear

system for.

Step 3) Compute and (independent of and ).Step 4) Compute and (dependent on and ) and

solve the linear systemfor .

During the initial stage of tuning the parameters and basedon the labeled image and the first unlabeled image, steps 1 and3 need to be done only once, whereas steps 2 and 4 have to berepeated. This is very favorable because, in our experiments,steps 1 and 3 are often more time consuming than steps 2 and4. If is fully labeled, then are known, andsteps 1 and 2 can be skipped. Starting from the second unlabeledimage, only steps 3 and 4 are performed, and no further tuningof parameters is done. Due to the constraint (4), can beobtained by without needing to solve any linearsystem.Let be the number of pixels in image . Let

and be the sizes of the geometric and photometricwindows, respectively. Let be the size of windowfor computing feature vectors. Let be the number ofphotometric neighbors selected within each 17 17 window.Let be the number of pixels in selected for computingacross-image photometric similarities. The costs of computingvarious matrices are1) : for ;2) : for ;3) : ;4) : .As for the cost of solving each linear system, our current im-plementation simply uses sparse Gaussian elimination, whosecost for each is in the worst case. However, due tothe facts that the matrices are structured and sparse with

number of nonzero entries, it is our optimismthat iterative solvers of much less complexity are possible. In-deed, in the single-image case, some efficient domain decom-position methods have been developed [13]. The model that wepropose can be used in conjunction with any linear solver. Sometimings of the various steps are reported in the next section.

III. EXPERIMENTAL RESULTS

In this section, we evaluate the performance of the proposedmultiple-image segmentation model empirically using threeimage data sets. All algorithms are implemented in MATLAB 7with a Core i7 3.07-GHz machine.

A. Image Data Sets

In the first experiment, a set of 58 breast cancer cell images ofvarious sizes in [17] is used. Overall, these im-ages have similar features. However, some contain benign cells,whereas some contain both benign and malignant cells, so thatthere are some variations among them. A manual segmentationof the cell nuclei is available in [17], which is used as the groundtruth to evaluate the accuracy of our method. In the second ex-periment, a set of 20 retinal images of size 292 283, whichis called DRIVE [18], is used. Some images show some sign

Fig. 7. Original image and its ground truth segmentation.

Fig. 8. Retinal layer images. Box plots of (a) the accuracy of the segmentationsand (b) the -measure for the ONL region retrieved by the proposed multiple-image model (MI) with and , SVM, KNN, KMeans, andSSKMeans.

of early diabetic retinopathy, whereas others do not. A manualsegmentation of the vasculature is available in [18]. In the thirdexperiment, a set of 30 cross sections of a human retina withsize 256 384 in [17] is used. Color distortion due to stainingand shape variation of the layers make the segmentation taskvery challenging. A manual segmentation of the layers is alsoavailable in [17].

B. Testing Method

For each experiment, one image is served as the labeledimage, and the remaining images are segmented using theproposed semisupervised multiple-image model (MI). For thethird experiment, only 50% of the labels are used for computing

(i.e., ). The parametersand for the similarity measures (13) and (14) are calibratedusing one labeled image and one unlabeled image. They arethen used throughout the whole image data set. Before seg-menting the images, a color transfer [19] is applied to eachunlabeled image so that its color histogram matches that of thelabeled image. This is to correct the difference in brightness andcontrast between images. To validate our method, we compareit with four other methods, namely, the classical support vectormachine (SVM), the -nearest neighbor (KNN), the -means(KMeans), and the semisupervised -means (SSKMeans) [20].The first two methods are supervised methods; the third isunsupervised; the fourth is semisupervised. For the supervised



Fig. 9. Segmentation of five of the testing images obtained by the proposed method (MI) and others. The ground truth segmentation in Fig. 7 is used as the trainingset for the MI, SVM, KNN, and SSKMeans methods.

and semisupervised methods, the labeled pixels are used as thetraining set. The accuracy, which is computed as the number ofcorrectly classified pixels divided by the total number of pixels,and the -measure, which is computed as the harmonic meanof precision and recall for the foreground object, i.e.,

precision recallprecision recall

are used as a measure of segmentation accuracy. The -mea-sure takes true positives, false positives, and false negatives intoaccount. A higher suggests a better performance. To furtherillustrate the proposed model, we compare it with a state-of-the-art blood vessel segmentation algorithm MLVESSEL [21],[22] using the DRIVE and the breast cancer cell data sets.

C. Performance

Breast cancer cell images: In this image data set, eachimage consists of two classes: background (includes stroma andwhite space) and cell nuclei. The labeled image is shown inFig. 1. The accuracy of the segmentations and the -measurefor the nucleus region obtained by the various methods are de-picted as box plots1 in Fig. 2(a) and (b), respectively. We ob-serve that the overall performance of the proposed method isthe best among all the methods. The segmentations of two ofthe testing images are shown in Fig. 3. The proposed methodis able to extract the nuclei accurately. It also gives smoother

1In the box plots, the upper, middle, and lower line of the box denote the 75%,50%, and 25% percentile value, respectively. The line above the box denotes the75% Percentile Value 1.5 Interquartile Range. The line below the box de-notes the 25% Percentile Value 1.5 Interquartile Range. The pluses denoteoutliers.



Fig. 10. Retinal images. Box plots of (a) the accuracy of the segmentationsand (b) the -measure for the blood vessels retrieved by the proposed multiple-image model (MI) with and MLVESSEL (MLV).

boundaries because of the regularization effect of the geometricsimilarity.

Retinal images: In this data image set, each image consistsof three classes: blood vessels, retina, and background. The la-beled image is shown in Fig. 4. The accuracy of the segmenta-tions and the -measure for the blood vessels obtained by thevarious methods are depicted as box plots in Fig. 5(a) and (b),respectively. The proposed method shows consistently higheraccuracy values. The segmentations of five randomly chosentest images are shown in Fig. 6. The proposed method is able toretrieve many of the blood vessels, whereas others give poorerresults.

Retinal layer images: In this image data set, each imageconsists of a background and four retinal layers: (from left toright) ganglion cell layer, inner nucleus layer, outer nucleuslayer (ONL), and rod outer segment. Understanding the mor-phological and cellular changes of a retina in response to dis-ease is crucial for diagnosis of retinal detachment. In particular,the extracted ONL layer is immediately useful for further anal-ysis such as nuclei counting. The labeled image is shown inFig. 7. The accuracy of the segmentations and the -measure forthe ONL region obtained by the various methods are depictedas box plots in Fig. 8(a) and (b), respectively. The proposedmethod shows higher accuracy values consistently. The seg-mentations of five randomly chosen testing images are shownin Fig. 9. The proposed method gives a high-quality segmenta-tion of the ONL, which could be used for further analysis.

Comparison to a state-of-the-art blood vessel segmen-tation algorithm: Here, we compare our algorithm with theMLVESSEL algorithm [21], [22], which is a state-of-the-artretinal blood vessel segmentation algorithm. This algorithmexploits domain knowledge of the retinal blood vessel and isthus expected to outperform our general-purpose algorithmwhen applied to segment retinal blood vessels. The accuracyand -measure with respect to the DRIVE data set are shownin Fig. 10. The proposed algorithm exhibits lower accuracy and-measure, but it is very competitive. The results with respect

to the breast cancer cell images are shown in Fig. 11. The pro-posed model outperforms MLVESSEL in this case. This showsthe usefulness of the proposed model as a general-purposemodel.

Computational efficiency: The CPU times (averaged overthe numerous images in the respective figure) required for thevarious steps outlined in Section II-E to obtain the results in

Fig. 11. Breast cancel cell images. Box plots of (a) the accuracy of the segmen-tations and (b) the -measure for the nucleus region retrieved by the proposedmultiple-image model (MI) with and MLVESSEL (MLV).

TABLE ICPU TIMES (AVERAGED OVER THE NUMEROUS IMAGES IN THERESPECTIVE FIGURE) REQUIRED FOR THE VARIOUS STEPS OF THE

PROPOSED MULTIPLE-IMAGE MODEL. FOR , THE AVERAGE OVERIS REPORTED

Fig. 12. Labeled image (part of a larger image) and its ground truth seg-mentation boundaries (green lines). The red and blue dots are SIFT keypointsserved as the photometric neighbor set for the nucleus part and the backgroundpart, respectively.

Figs. 3, 6, 9 are reported in Table I. Since is fully labeled, thecomputation of , , and is not required. We re-mark that a direct comparison between the times for computing

and the times for computing is inappropriate.This is because the linear solver we used for is aMATLABbuilt-in function, which is highly optimized and is a compiledcode.

IV. PARAMETER ANALYSIS

In this section, we empirically study the effect of various pa-rameters on the proposed method. Specifically, we study thechoice of photometric neighbors and the brightness of the sim-ulated structure due to manifestations of disease.

A. Across-Image Photometric Neighbor Selection

Due to the relatively large size of images, it can be desirable touse only a subset of labeled pixels for computing across-imagephotometric similarity to speed up the process.



Fig. 13. Segmentation of benign images. (First column) Retrieved nuclei and (second column) background obtained by the proposed multiple-image model using(first row) SIFT keypoints, (second row) randomly selected pixels (0.2% of pixels in ), and (third row) all the pixels. The red box indicates the location of thelabeled part of the image.

1) Images: In this experiment, we aim to study the tradeoffbetween the computation time and the quality of the segmen-tation result. Two breast cancer cell images (benign and malig-nant) of size 896 768 in [17] are used.

2) Method: For the benign (respectively malignant) image,a rectangular patch with size 205 227 (respectively 240222) has been labeled. To segment the image, we consider

its labeled part as (i.e., ) and the whole



Fig. 14. Segmentation of malignant images. (First column) Retrieved nuclei and (second column) background obtained by the proposed multiple-image modelusing (first row) SIFT keypoints, (second row) randomly selected pixels (0.2% of pixels in ), and (third row) all the pixels. The red box indicates the locationof the labeled part of the image.

image itself as with labeled pixels . We compute theacross-image similarity using

(i) a set of keypoints of shown in Fig. 12 obtained by thescale invariant feature transform (SIFT) keypoint detectionmethod [23];

(ii) a set of randomly selected labeled pixels in ; and(iii) all the pixels in .

SIFT was designed such that only high discriminative pointswere selected. The two images have 142 and 207 keypointsdetected, respectively, which account for about 0.3% of the



number of pixels in . For (ii), about 0.2% of the number ofpixels in are selected.3) Results: The segmentation result of the benign image and

that of the malignant image are shown in Figs. 13 and 14, re-spectively. To assess the performance, we use (iii) as the base-line and compute the percentage symmetric difference betweenthe results of (i) [respectively (ii)] and (iii). Both (i) and (ii) haveless than 3% difference for both images. For (ii), we have alsotried using various percentages of selected points, from 0.2%to 100%. As expected, the percentage symmetric difference de-creases from 3% to 0% (results not shown). Concerning timeconsumption, both (i) and (ii) run much faster than (iii) (about3–5 times faster). The running time of both approaches (i) and(ii) are almost the same when the same numbers of pixels areselected for both (i) and (ii) as the time spent in SIFT keypointdetection is negligible. We show that using a subset of labeledpixels can speed up the segmentation process while still ob-taining a reasonably good performance in this set of image.

B. Simulated Manifestations of Disease

In the presence of retinal disease, some hemorrhages and le-sions, which are usually yellowish and reddish objects, may befound in a retinal image. In this experiment, we aim to test therobustness of the proposed method to such kinds of alien ob-jects. The primary objective here is to extract the blood vessels.Hence, it is hoped that these alien objects are put into the back-ground and do not hinder the identification of the blood vessels.1) Images: Two retinal images in [18] are used. The first one

(shown in Fig. 4) is served as the labeled image. The secondone (not shown) is used to generate two sets of ten simulateddisease images. In the first (respectively second) set of images,which are referred to as the yellowish (respectively reddish) set,a circular disc with an increasing degree of yellowness (respec-tively redness) is juxtaposed to the center of each image. Three(respectively two) of the simulated images in the yellowish (re-spectively reddish) set are shown in the first row in Fig. 15(a)[respectively Fig. 15(b)].2) Method: The 20 simulated images are segmented one at a

time using the proposed multiple-image model, with the imagein Fig. 4 a fully labeled sample.3) Results: The segmentations of three of the simulated

images in the yellowish set are shown in the second row inFig. 15(a). The accuracy of the segmentations is shown inFig. 16(a). We observe that the proposed method is able toclassify much of the alien object as retina, and the accuracy isnot affected at all. This is mainly due to the high color contrastbetween the yellowish alien object and the blood vessels anddue to the presence of the yellowish optic disc on the right inthe training data.The segmentations of two of the simulated images in the red-

dish set are shown in the second row in Fig. 15(b). The accuracyof the segmentations is shown in Fig. 16(b). Observe that theaccuracy drops between Image 4 and Image 5. The decrease inthe accuracy is mainly due to the misclassification of pixels inthe reddish circular disc. For the first four images, the proposedmethod is able to classify much of the alien object as retina. Itshows that the proposed method has a certain degree of toler-ance. However, when the intensity of the alien object becomes

Fig. 15. Segmentation of three simulated images with (a) a yellowish objectand (b) a reddish object obtained by the proposed multiple-image model

.

Fig. 16. Simulated retinal disease image [(a) yellowish and (b) reddish]. Ac-curacy of the segmentations obtained by the proposed multiple-image model

.

too close to that of the blood vessel pixels (i.e., Image 5 to Image10), the object is classified as blood vessel by the properties ofthe proposed model.

V. CONCLUSION

In this paper, we have proposed and developed a semiauto-matic optimization model for segmentation of multiple images.The model has a quadratic objective function and linear con-straints. Due to the discrete maximum/minimum principles, theoptimality conditions simply boil down to solving linear sys-tems (as opposed to the nonlinear Karush–Kuhn–Tucker sys-tems for general quadratic programming problems). In our ap-plications, the two parameters can be easily tuned. Once initialtuning is done, the setup can be used to segment all other imageswithin the collection automatically. The quality of the results is



also high. However, it relies on the logical assumption that thedifferent classes can be separated in the feature space and thatthe user-supplied samples can represent each class well. We re-mark that, while we used a 27-D feature space derived fromRGBvalues, any other feature space that can separate the classescan be used.

APPENDIX

In this Appendix, we derive the optimality conditions forthe constrained problem defined by (2)–(5). Theorem 1 givesthe optimality conditions when the bilateral constraints (3) aredropped. Then, in Theorem 2, we show that the solution inTheorem 1 satisfies the weak discrete maximum and minimumprinciples so that the bilateral constraints (3) are automaticallysatisfied (Theorem 3). Uniqueness of solution is given inTheorem 4.We first introduce some notations, in addition to the ones in

(6)–(9) and (15)–(17). Let be the down-sampling matrixfrom to . Let and .Then, upon reordering of the variables, each can be parti-tioned as

The matrix can be also accordingly partitioned as

Other variables with subscripts and/or are similarlydefined. The constraints (4) and (5) can be expressed as

and , respectively.

A. Optimality Conditions

Theorem 1: The optimality conditions of the problem

subject to

for and are given by

for

Proof: The Lagrangian function for the optimizationproblem is given by

where and are Lagrange multipliers for the constraints.The optimality conditions are given by

for and . In particular, we have

(18)

By summing the above equations for andusing , we have

Thus, for . Hence, (18) becomes

We show in Theorem 5 that the submatrix

is nonsingular. Thus, we have . Together with theconstraints , the optimality conditions become

(19)That is, .

B. Discrete Maximum/Minimum Principle

A solution to the equation is said to satisfythe weak discrete maximum principle if

The weak discrete minimum principle is similarly defined.Let and be two pixels. Pixel is said to

be path connected to pixel if there exists a chain of pixelssuch that , , and each

is a neighbor of , i.e., . Notethat, due to the use of geometric neighbor, every pixel ispath connected to some pixel in if . However, sincecan be empty in our setting, we shall assume that every pixel

in is path connected to some pixels in . Thissimply requires that each pixel in has a photometric neighborin , which is how we defined in (12).Theorem 2: Assume is path connected to .

The solution of satisfies the weak maximum andminimum principles.



Proof: Suppose that there exists such that. Then, we have . Since , we

have

so that is a weighted average of its neighboringvalues and is greater than or equal to each of them. This im-plies that for all . Repeating

this argument to each and using the assump-tion that is path connected to some , we conclude that

. The weak discrete minimumprinciple can be proved by simply replacing the maximum withthe minimum.Theorem 3: Assume that both and

are nonempty sets, i.e., at least one of the givenlabels is class and at least one of the given labels is not class. The solution of satisfies the constraints

.Proof: Under the assumptions and boundary conditions

(5), we have

By the weak maximum and minimum principles, we have

C. Uniqueness of the Solution

Theorem 4: The solution of is unique.Proof: Suppose that . Then, we have

By the weak maximum and minimum principles, we have

Thus, is nonsingular.Theorem 5: The matrix

is nonsingular.Proof: From (19), we note that, up to a reordering of the

pixels, we have

Thus, . By Theorem 4,so that too.

REFERENCES

[1] Y. Boykov and V. Kolmogorov, “An experimental comparison ofmin-cut/max-flow algorithms for energy minimization in vision,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 9, pp. 1124–1137,Sep. 2004.

[2] J. Guan and G. Qiu, “Interactive image matting using optimization,”Sch. Comput. Sci. Inf. Technol., Univ. Nottingham, University Park,U.K., Tech. Rep. 01-2006, 2006.

[3] C. Rother, V. Kolmogorov, and A. Blake, “Grabcut: Interactive fore-ground extraction using iterated graph cuts,” ACM Trans. Graph., vol.23, no. 3, pp. 309–314, Aug. 2004.

[4] J. Wang andM. Cohen, “An iterative optimization approach for unifiedimage segmentation and matting,” in Proc. IEEE Int. Conf. Comput.Vis., 2005, pp. 936–943.

[5] S. Yu and J. Shi, “Segmentation given partial grouping constraints,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2, pp. 173–183,Feb. 2004.

[6] G. Gilboa and S. Osher, “Nonlocal linear image regularization and su-pervised segmentation,” Multiscale Model. Simul., vol. 6, no. 2, pp.595–630, 2007.

[7] Y. Boykov and M. P. Jolly, “Interactive graph cuts for optimalboundary and region segmentation of objects in -d images,” in Proc.IEEE Int Conf. Comput. Vis., 2001, pp. 105–112.

[8] Y. Li, J. Sun, C. K. Tang, and H. Y. Shum, “Lazy snapping,” ACMTrans. Graph., vol. 23, no. 3, pp. 303–308, Aug. 2004.

[9] A. Blake, C. Rother, M. Brown, P. Perez, and P. Torr, “Interactiveimage segmentation using an adaptive GMMRF model,” in Proc.ECCV, 2004, pp. 428–441.

[10] A. Protiere and G. Sapiro, “Interactive image segmentation via adap-tive weighted distances,” IEEE Trans. Image Process., vol. 16, no. 4,pp. 1046–1057, Apr. 2007.

[11] N. Houhou, X. Bresson, A. Szlam, T. F. Chan, and J.-P. Thiran, “Semi-supervised segmentation based on non-local continuous min-cut,” inProc. SSVM, 2009, pp. 112–123.

[12] J. Guan and G. Qiu, “Interactive image segmentation using optimiza-tion with statistical priors,” in Proc. ECCV, Graz, Austria, 2006.

[13] M. K. Ng, G. Qiu, and A. M. Yip, “Numerical methods for interac-tive multiple-class image segmentation problems,” Int. J. Imag. Syst.Technol., vol. 20, no. 3, pp. 191–201, Sep. 2010.

[14] F. Wang, C. Zhang, H. Shen, and J. Wang, “Semi-supervised classifi-cation using linear neighborhood propagation,” in Proc. CVPR, 2006,pp. 160–167.

[15] O. Delalleau, Y. Bengio, and N. Le Roux, “Efficient non-parametricfunction induction in semi-supervised learning,” in Proc. 10th Int.Workshop AI Stat., 2005, pp. 96–103.

[16] J. Guan and G. Qiu, “Modeling user feedback using a hierarchicalgraphical model for interactive image retrieval,” in Advances in Mul-timedia Information Processing, ser. Lecture Notes in Computer Sci-ence, H. H.-S. Ip, H. Leung, O. C. Au, M.-T. Sun,W.-Y. Ma, and S.-M.Hu, Eds. Berlin, Germany: Springer-Verlag, 2007, pp. 18–29.

[17] E. Gelasca, B. Obara, D. Fedorov, K. Kvilekval, and B. Manjunath,“A biosegmentation benchmark for evaluation of bioimage analysismethods,” BMC Bioinformat., vol. 10, p. 368, 2009.

[18] J. Staal, M. Abramoff, M. Niemeijer, M. Viergever, and B. Ginneken,“Ridge based vessel segmentation in color images of the retina,” IEEETrans. Med. Imag., vol. 23, no. 4, pp. 501–509, Apr. 2004.

[19] F. Pitié, A. Kokaram, and R. Dahyot, “Automated colour grading usingcolor distribution transfer,” Comput. Vis. Image Underst., vol. 107, no.1/2, pp. 123–137, Jul. 2007.

[20] S. Basu, A. Banerjee, and R. Mooney, “Semi-supervised clustering byseeding,” in Proc. ICML, 2002, pp. 27–34.

[21] J. Soares, J. Leandro, R. Cesar, H. Jelinek, andM. Cree, “Retinal vesselsegmentation using the 2-D Gabor wavelet and supervised classifica-tion,” IEEE Trans. Med. Imag., vol. 25, no. 9, pp. 1214–1222, Sep.2006.

[22] J. Soares and R. Cesar, Automated Image Detection of RetinalPathology. Boca Raton, FL: CRC Press, 2009, pp. 221–269.

[23] D. Lowe, “Distinctive image features from scale-invariant keypoints,”Intl. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.



Yan Nei Law received the Ph.D. degree in computerscience from the University of California, Los An-geles, in 2006.She joined the Bioinformatics Institute, Singapore,

in 2006 as a Postdoctoral Research Fellow. Her re-search interests include bioimaging with emphasis ondeveloping image processing and data mining tech-niques for high throughput images.

Hwee Kuan Lee received the Ph.D. degree in the-oretical physics from Carnegie Mellon University,Pittsburgh, PA, in 2001, where he worked on MonteCarlo simulations of liquid–liquid phase transitionsand quasi-crystals.He is a Principal Investigator with the Imaging In-

formatics division, Bioinformatics Institute, Singa-pore. His current research work involves developingand deployment of computer vision algorithms forhigh throughput bioimaging experiments. He held ajoint postdoctoral position with the Oak Ridge Na-

tional Laboratory, Oak Ridge, TN, and the University of Georgia, Athens, wherehe worked on developing advanced Monte Carlo methods and nanomagnetism.In 2003, with an award from the Japan Society for Promotion of Science, hemoved to Tokyo Metropolitan University, where he developed solutions to ex-tremely long time scaled problems and a reweightingmethod for nonequilibriumsystems. In 2005, he joined Data Storage Institute, investigating novel recordingmethods such as hard disk recording via magnetic resonance. In 2006, he joinedthe Bioinformatics Institute as a Principal Investigator with the Imaging Infor-matics division.

Michael K. Ng received the B.Sc. and M.Phil.degrees from the University of Hong Kong, HongKong, in 1990 and 1992, respectively, and the Ph.D.degree from Chinese University of Hong Kong,Shatin, Hong Kong, in 1995.He is a Professor with the Department of Math-

ematics, Hong Kong Baptist University, Kowloon,Hong Kong. From 1995 to 1997, he was a ResearchFellow of Computer Sciences Laboratory with Aus-tralian National University. From 1997 to 2005, hewas an Assistant/Associate Professor with the Uni-

versity of Hong Kong before joining Hong Kong Baptist University. His re-search interests include bioinformatics, data mining, image processing, scien-tific computing and data mining.Dr. Ng serves on the editorial boards of several international journals. See

http://www.math.hkbu.edu.hk/~mng

Andy M. Yip received the B.Sc. degree in math-ematics from Chinese University of Hong Kong,Shatin, Hong Kong, in 1998, the M.Phil. degree inmathematics from the University of Hong Kong,Hong Kong, in 2000, and the Ph.D. degree inmathematics from the University of California, LosAngeles, in 2005.In July 2005, he joined the Department of

Mathematics, National University of Singapore,Singapore, as an Assistant Professor. His researchinterests include variational and PDE methods in

image processing and data mining algorithms.


Documents

A semisupervised segmentation model for collections of images.bak