NAFIPS 2005 - 2005 Annual Meeting of the North American Fuzzy Information Processing Society

Fuzzy Clustering in Parallel Universes

Bernd Wiswedel and Michael R. BertholdDepartment of Computer and Information Science, University of Konstanz

78457 Konstanz, Germany{wiswedel,berthold}@inf.uni-konstanz .de

Abstract-We propose a modified fuzzy c-Means algorithmthat operates on different feature spaces, so-called paralleluniverses, simultaneously. The method assigns membership val-ues of patterns to different universes, which are then adoptedthroughout the training. This leads to better clustering resultssince patterns not contributing to clustering in a universe are(completely or partially) ignored. The outcome of the algorithmare clusters distributed over different parallel universes, eachmodeling a particular, potentially overlapping, subset of thedata. One potential target application of the proposed method isbiological data analysis where different descriptors for moleculesare available but none of them by itself shows global satisfactoryprediction results. In this paper we show how the fuzzy c-Meansalgorithm can be extended to operate in parallel universes andillustrate the usefulness of this method using results on artificialdata sets.

I. INTRODUCTION

In recent years, researchers have worked extensively in thefield of cluster analysis, which has resulted in a wide range of(fuzzy) clustering algorithms [1], [2]. Most of the methods as-sume the data to be given in a single (mostly high-dimensionalnumeric) feature space. In some applications, however, it iscommon to have multiple representations of the data available.Such applications include biological data analysis, in which,e. g. molecular similarity can be defined in various ways.Fingerprints are the most commonly used similarity measure.A fingerprint in a molecular sense is a binary vector, wherebyeach bit indicates the presence or absence of a molecularfeature. The similarity of two compounds can be expressedbased on their bit vectors using the Tanimoto coefficient forexample. Other descriptors encode numerical features derivedfrom 3D maps, incorporating the molecular size and shape,hydrophilic and hydrophobic regions quantification, surfacecharge distribution, etc. [3]. Further similarities involve thecomparison of chemical graphs, inter-atomic distances, andmolecular field descriptors. However, it has been shown thatoften a single descriptor fails to show satisfactory predictionresults [4].

Other application domains include web mining where adocument can be described based on its content and on anchortexts of hyperlinks pointing to it [5]. Parts in CAD-cataloguescan be represented by 3D models, polygon meshes or textualdescriptions. Image descriptors can rely on textual keywords,color information, or other properties [6].

In the following we denote these multiple representations,i. e. different descriptor spaces, as Parallel Universes [7], eachof which having representations of all objects of the data set.

The challenge that we are facing here is to take advantageof the information encoded in the different universes to findclusters that reside in one or more universes each modelingone particular subset of the data. In this paper, we develop anextended fuzzy c-Means (FCM) algorithm [8] that is applicableto parallel universes, by assigning membership values fromobjects to universes. The optimization of the objective functionis similar to the original FCM but also includes the learningof the membership values to compute the impact of objects touniverses.

In the next section, we will discuss in more detail theconcept of parallel universes; section III presents relatedwork. We formulate our new clustering scheme in section IVand illustrate its usefulness with some numeric examples insection V.

II. PARALLEL UNIVERSESWe consider parallel universes to be a set of feature

spaces for a given set of objects. Each object is assigneda representation in each single universe. Typically, paralleluniverses encode different properties of the data and thus leadto different measures of similarity. (For instance, similarityof molecular compounds can be based on surface chargedistribution or fingerprint representation.) Note, due to theseindividual measurements they can also show different struc-tural information and therefore exhibit distinctive clustering.This property differs from the problem setting in the so-calledMulti-Vew Clustering [9] where a single universe, i. e. view,suffices for learning but the aim is on binding different viewsto improve the classification accuracy and/or accelerating thelearning process. The objective for our problem definition is onidentifying clusters located in different universes whereby eachcluster models a subset of the data based on some underlyingproperty.

Since standard clustering techniques are not able to copewith parallel universes, one could either restrict the analysis toa single universe at a time or define a descriptor space compris-ing all universes. However, using only one particular universeomits information encoded in the other representations andthe construction of a joint feature space and the derivation ofan appropriate distance measure are cumbersome and requiregreat care as it can introduce artifacts.

III. RELATED WORKClustering in parallel universes is a relatively new field

of research. In [6], the DBSCAN algorithm is extended and

0-7803-9187-X/051$20.00 ©2005 IEEE. 567

applied to parallel universes. DBSCAN uses the notion ofdense regions by means of core objects, i. e. objects that havea minimum number k of objects in their (e-) neighborhood. Acluster is then defined as a set of (connected) dense regions.The authors extend this concept in two different ways: Theydefine an object as a neighbor of a core object if it is in thee-neighborhood of this core object either (1) in any of therepresentations or (2) in all of them. The cluster size is finallydetermined through appropriate values of e and k. Case (1)seems rather weak, having objects in one cluster even thoughthey might not be similar in any of the representational featurespaces. Case (2), in comparison, is very conservative since itdoes not reveal local clusters, i. e. subsets of the data thatonly group in a single universe. However, the results in [6]are promising.

Another clustering scheme called "Collaborative fuzzy clus-tering" is based on the FCM algorithm and was introducedin [10]. The author proposes an architecture in which objectsdescribed in parallel universes can be processed togetherwith the objective of finding structures that are common toall universes. Clustering is carried out by applying the c-Means algorithm to all universes individually and then byexchanging information from the local clustering results basedon the partitioning matrices. Note, the objective function, asintroduced in [10], assumes the same number of clusters ineach universe and, moreover, a global order on the clusterswhich-in our opinion-is very restrictive due to the randominitialization of FCM.

A supervised clustering technique for parallel universes wasgiven in [7]. It focuses on a model for a particular (minor) classof interest by constructing local neighborhood histograms,so-called Neighborgrams for each object of interest in eachuniverse. The algorithm assigns a quality value to each Neigh-borgram and greedily includes the best Neighborgram, nomatter from which universe it stems, in the global predictionmodel. Objects that are covered by this Neighborgram arefinally removed from consideration in a sequential coveringmanner. This process is repeated until the global model hassufficient predictive power.

Blum and Mitchell [5] introduced co-training as a semi-supervised procedure whereby two different hypotheses aretrained on two distinct representations and then bootstrap eachother. In particular they consider the problem of classifyingweb pages based on the document itself and on anchor texts ofinbound hyperlinks. They require a conditional independenceof both universes and state that each representation shouldsuffice for learning if enough labeled data were available. Thebenefit of their strategy is that (inexpensive) unlabeled dataaugment the (expensive) labeled data by using the predictionin one universe to support the decision making in the other.

Other related work includes reinforcement clustering [11]and extensions of partitioning methods-such as k-Means, k-Medoids, and EM-and hierarchical, agglomerative methods,all in [9].

IV. CLUSTERING ALGORITHM

In this section, we introduce all necessary notation, reviewthe FCM algorithm and formulate a new objective functionthat is suitable to be used for parallel universes. The technicaldetails, i. e. the derivation of the objective function, can befound in the appendix section.

In the following, we consider IUI, 1 < u < IUI,parallel universe, each having representational feature vec-tors for all objects - = (xi . .* ,u,Awith Au the dimensionality of the u-th universe. We de-pict the overall number of objects as ITI, 1 < i < ITI.We are interested in identifying cu clusters in universeu. We further assume appropriate definitions of distancefunctions for each universe du (Zk,u, 5.) 2 where Wk,' -(Wk,u, X.. Wk,ua ... Wk,u,Au) denotes the k-th prototype inthe u-th universe.We confine ourselves to the Euclidean distance in the

following. In general, there are no restrictions to the distancemetrics other than the differentiability. In particular, they donot need to be of the same type in all universes. This isimportant to note, since we can use the proposed algorithmin the same feature space, i. e. Xi,uL = Xi for any u1 andU2, but different distance measure across the universes.

A. Formulation of new objective functionThe original FCM algorithm relies on one feature space only

and minimizes the objective function as follows. Note that weomit the subscript u here as we consider only one universe:

ITI c

Jm ZZ vd d (̀3k 5i)2i=1 k=1

m E (1, oo) is a fuzzyfication parameter, and Vi,k the respec-tive value from the partition matrix, i. e. the degree to whichpattern xi belongs to cluster k. This function is subject tominimization under the constraint

c

Vi: EVi,k - 1,k=1

requiring that the coverage of any pattern i needs to accumu-late to 1.The above objective function assumes all cluster candidates

to be located in the same feature space and is therefore notdirectly applicable to parallel universes. To overcome this,we introduce a matrix zi,u, 1 < i < ITI, 1 < u < UI,encoding the membership of patterns to universes. A valuez,u close to 1 denotes a strong contribution of pattern xi tothe clustering in universe u, and a smaller value, a respectivelylesser degree. Zi,u has to satisfy standard requirements formembership degrees: it must accumulate to 1 considering alluniverses and must be in the unit interval.The new objective function is given with

ITI jUl c,='_ :F n, v Vm 2.\Jm,n Z Zi, Vi,k,udu (Wk,u, Xi,u)

i=1 u=l k=l(1)

568

Parameter n E (1, oc) allows (analogous to m) to have impacton the fuzzyfication of zi,u: The larger n the more equal thedistribution of zi,u, giving each pattern an equal impact to alluniverses. A value close to 1 will strengthen the compositionof zi,u and assign high values to universes where a patternshows good clustering behavior and small values to thosewhere it does not. Note, we now have JUI different partitionmatrices (v) to assign membership degrees of objects to clusterprototypes.As in the standard FCM algorithm, the objective function

has to fulfill side constraints. The coverage of a pattern amongthe partitions in each universe must accumulate to 1:

C,

Vi.u:EVi,ku =1. (2)k14_

Additionally, as mentioned above, the membership of a patternto different universes has to be in total 1, i.e.

luiV :E Z'U 1. (3)

u=1

The minimization is done with respect to the parametersVi,k,u, zi,u, and Wk,u Since the derivation of the objectivefunction is more of technical interest, please refer to theappendix for details.The optimization splits into three parts. The optimization of

the partition values Vi.k,u for each universe; determining themembership degrees of patterns to universes zi,u and finallythe adaption of the center vectors of the cluster representativesWk,u.The update equations of these parameters are given in (4),

(5), and (6). For the partition values Vi,k,u, it follows

Vi,k,u 1Cu -, .2 \nm-du(W-k, u,X7,Uk=l e(ku zt

Note, this equation is independent of the values Zzi, andis therefore identical to the update expression in the singleuniverse FCM. The optimization with respect to Zzi, yields

-i'U (5)E-.k= i!k,4udu(W-k,uiXi,u)

=1VECk=1 i7,k,ud(k72 Xu

and update equation for the adaption of the prototype vectorsWik,2 is of the form

EITI Z%m'XU.Wk,u,a - L.1ZiVikuXua (6)wk,u,a= ITI n m(6

Li=l Zz, Vi,k,uThus, the update of the prototypes depends not only on thepartitioning value Vi,k,u, i. e. the degree to which pattern ibelongs to cluster k in universe u, but also to z,u representingthe membership degrees of patterns to the current universe ofinterest. Patterns with larger values zi,u will contribute moreto the adaption of the prototype vectors, while patterns with asmaller degree accordingly to a lesser extent.

Equipped with these update equations, we can introduce theoverall clustering scheme in the next section.

B. Clustering algorithmSimilar to the standard FCM algorithm, clustering is carried

out in an iterative manner, involving three steps:1) Update of the partition matrices (v)2) Update of the membership degrees (z)3) Update of the prototypes (wi)

More precisely, the clustering procedure is given as:

(1) Given: Input pattern set described in JUI paralleluniverses: x 1 < i < ITI, 1 < u < JUI

(2) Select: A set of distance metrics du (., .)2, and thenumber of clusters for each universe ca, 1 < u < IU ,define parameter m and n

(3) Initiate: Partition matrices Vi,k,u with random valuesand the cluster prototypes by drawing samples fromthe data. Assign equal weight to all membershipdegrees zi,u-

(4) Train:(5) Repeat(6) Update partitioning values Vi,k,u according to (4)(7) Update membership degrees Zi,u according to (5)(8) Compute prototypes wih,u using (6)(9) until a termination criterion has been satisfied

The algorithm starts with a given set of universe definitionsand the specification of the distance metrics to use. Also, thenumber of clusters in each universe needs to be defined inadvance. The membership degrees zi, are initialized withequal weight (line (3)), thus having the same impact on alluniverses. The optimization phase in line (5) to (9) is-incomparison to the standard FCM algorithm-extended by theoptimization of the membership degrees, line (7). The possi-bilities for the termination criterion in line (9) are manifold.One can stop after a certain number of iterations or use thechange of the value of the objective function (1) betweentwo successive iterations as stopping criteria. There are alsomore sophisticated approaches, for instance the change to thepartition matrices during the optimization.

Just like the FCM algorithm, this method suffers from thefact that the user has to specify the number of prototypes to befound. Furthermore, our approach even requires the definitionof cluster counts per universe. There are numerous approachesto suggest the number of clusters in the case of the standardFCM, [12], [13] to name but a few. Although we have notyet studied their applicability to our problem definition we dobelieve that some of them can be adapted to be used in ourcontext as well.

V. EXPERIMENTAL RESULTSIn order to demonstrate this approach, we generated syn-

thetic data sets with different numbers of parallel universes.For simplicity we restricted the size of a universe to 2dimensions and generated 2 Gaussian distributed clusters

569

Untverse 2

0.2 0.4 0.6 0.8

Universe I

0.8

1 0 0.2 0.4 0.6 0.8

Universe 2

0.8

0U0.2 0.4 0.6

Universe 3

0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.6 1 0 0.2 0.4 0.6

Fig. 1. Three universes of a synthetic data set. The top figures show only objects that were generated within the respective universe (using two clusters peruniverse). The bottom figures show all patterns; note that most of them (i. e. the ones from the other two universes), are noise in this particular universe. Forclarification we use different shapes for objects that origin from different universes.

(per universe). We then assigned each object to one of theuniverses and drew its features in that universe according tothe distribution of the cluster (randomly picking one of thetwo). The features of this object in the other universes were

drawn from a uniform distribution, i. e. they represent noisein these universes. Figure I shows an example data set withthree universes. The top figures show only the objects thatwere generated to cluster in the respective universe, i. e. theydefine the reference clustering. The bottom figures include allobjects and show the universes as they are presented to theclustering algorithm. Approximately 2/3 of the data do notcontribute to clustering in a universe and therefore are noise.To compare the results we applied the standard FCM

algorithm to the joint feature space of all universes andset the number of desired clusters to the overall numberof generated clusters. Thus, the numbers of dimensions andclusters were two times the number of universes.We forceda crisp cluster membership decision based on the highestvalue of the partition values, i. e. the cluster to a pattern i isdetermined by k = argmaxl<k<c{vi,k}. When the universeinformation was taken into account, a cluster decision isbased on the highest value of Zi,u Vi,k,u. Thus, universe andcluster index (u, k) for pattern i are computed as (ii, k) =

argmax,s<uIul {Zi,u * Vi,k,u}.

We used the following quality measure to compare differentclustering results [6]:

QK(C) = E ICI (1 entropyK(Ci))cie TI

where K is the reference clustering, i. e. the clusters as

generated, C the clustering to evaluate, and entropyK(Ci)

0,6 * Joint Feature Space

0,5

0,3

0,1

0

2 3 4 5 6Number of Universes

Fig. 2. Clustering quality for 5 different data sets. The number of universesranges from 2 to 6 universes. Note how the cluster quality of the jointfeature space drops sharply whereas the parallel universe approach seemsless affected. An overall decline of cluster quality is to be expected since thenumber of clusters to be detected increases.

the entropy of cluster Ci with respect to K. This function is1 if C equals K and 0 if all clusters are completely puzzledsuch that they all contain an equal fraction of the clusters inK. Thus, the higher the value, the better the clustering.

Figure 2 summarizes the quality values for 5 experimentscompared to the standard FCM. The number of clusters ranges

from 2 to 6. Clearly, for this data set, our algorithm takesadvantage of the information encoded in different universesand identifies the major parts of the original clusters. Obvi-ously this is by no means proof that the method will alwaysdetect clusters spread out over parallel universes but these earlyresults are quite promising.

570

0.8

0.6

0.4

0.2

0

0.8

0.6

0.4

0.2

D:6:. s = . . .

.62- r D = . a = .

v . O 0

,~~~~~~ ~:0.8 t

Unverse I Universe 3I t

O.

0

VI. CONCLUSION

We considered the problem of unsupervised clustering inparallel universes, i.e. problems where multiple representa-tions are available for each object. We developed an extensionof the fuzzy c-Means algorithm that uses membership degreesto model the impact of objects to the clustering in a particularuniverse. By incorporating these membership values into theobjective function, we were able to derive update equationswhich minimize the objective with respect to these values,the partition matrices, and the prototype center vectors. Theclustering algorithm works in an iterative manner using theseequations to compute a (local) minimum. The result areclusters located in different parallel universes, each modelingonly a subset of the overall data and ignoring data that do notcontribute to clustering in a universe.We demonstrated that the algorithm performs well on a

synthetic data set and exploits the information of havingdifferent universes nicely. Further studies will concentrate onthe applicability of the proposed method to real world data,heuristics that adjust the number of clusters per universe, andthe influence of noisy data.

ACKNOWLEDGMENT

This work was partially supported by the Research TrainingGroup 1024 funded by the Deutsche Forschungsgemeinschaft(DFG).

APPENDIX

In order to compute a minimum of the objective function (1)with respect to (2) and (3), we exploit a Lagrange technique tomerge the constrained part of the optimization problem withthe unconstrained one. This leads to a new objective functionFi that we minimize for each pattern xi individually,

Jul C,

Fi - 2_ Z,u i1,k,d Wtk, Xi,uu=1 k=1

lUI cu \ UI+ U - Vi,k,U) + A -E Zi,U .(7)

U=1 k=1 U=1

The parameters A and pau, 1 < u < IU, denote the Lagrangemultiplier to take (2) and (3) into account. The necessaryconditions leading to local minima of Fi read as

- 0,aZZ.i,u

OFi aFi aF.-0O -=O - =0aVi,k,u &A 'mu

1 < U< lull 1 < k < cu.

(8)

In the following we will derive update equations for the z andv parameters. Evaluating the first derivative of the equationsin (8) yields the expression

')F c11OF~ ftZ7; vm,kudu (tU, ,~~2-A 0,=n zu E ik U Wk Xi )2- =

wi,u ~k=1

AX n-1

zi u-= n * (9)

We can rewrite the above equation

From the derivative of F2 w. r. t. A in (8), it follows

-~. - 1

Zi,U Wkuu Xiu

k=1

zi,u 1

(10)

(11)

which returns the normalization condition as in (3). Using theformula for zi,U in (9) and integrating it into expression (11)we compute

E knu=l

(nj u=l

(Z%1~=i 1t

)-1

Ek-l i,k,uu (Wk,u, Xi,u)

= 1.(12)

We make use of (10) and substitute (Q) "' in (12). Note, weuse ii as parameter index of the sum to address the fact thatit covers all universes, whereas u denotes the current universeof interest. It follows

1 - Zi,u t 1(zkk,uWk,u:Xi,u,) )

k=1

zi - 14

ui=l (1k=l i,knuu Wk,ii, Xi,u))which can be simplified to

I _ iUI {EkU 1 vi,k,udu ( #k,u, X__i,u)Au= k 1 i,k, u vkuX

and retums anifrmediate update expression for the member-shipzv,u of pattern i to universe u (see also (5)):

Zi,U = ( 4 ) -

tE _- d',kUd(7k,UYi'i1 )2U- 2ku 1 i'k u(kixo)

Analogous to the calculations above we can derive the up-date equation for value Vi,k,uwhich represents the partitioningvalue of pattern i to cluster k in universe u. From (8) it follows

OFi = Z!t mvmM-1du (Wk ,XzuJ` 2

571

and hence1

1C, vm du(--# -# )2Ek=l i,k,u Wk,u, Xi,u

1

1-

C, vm du(--# - )2Ek=l i,k u Wk,u, Xi.u

and thus

Vm Z!n

z,k,u- km Tiu u Wk,u: Xi,u)

)1

m.,-1

,(13)

(14)

Zeroing the derivative of Fi w. r. t. I,,u will result in condi-tion (2), ensuring that the partition values sum to 1, i. e.

-Z1-E Vi,k,u = (15)0/lu k=1

We use (13) and (15) to come up with

Tu z! dn -W1-1.(m= ziudjk,u, Xi,u)2)

\u InL-I Cu1 \ /-1

1 (m/ Sug) 1 du(d(kj,uX Xi,U)) .(6)Equation (14) allows us to replace the first multiplier in (16).We will use the k notation to point out that the sum in (16)considers all partitions in a universe and k to denote oneparticular cluster coming from (13),

1 = tVi,k,u (cA (tk,u,z ,u) )

(P4U,-,U)2)r-

1 Vi= kv E (du (Wk1u: diu) )k=l du (W,,-,u)2

Vi,k,uWk

1

E d(w ax u) 2

d=d2wkuf,xi,u

For the sake of completeness we also derive the update rulesfor the cluster prototypesfrVk,u. We confine ourselves to theEuclidean distance here, assuming the data is normalized1:

Au 2

df (thk,U,c,o)p Zk (Wk,u,a -Xin ura) (17)

a=l

1 The derivation of the updates using other than the Euclidean distanceworks in a similar manner.

with Au the number of dimensions in universe u and Wk,u,athe value of the prototype in dimension a. Xi,u,a is the valueof the a-th attribute of pattern i in universe u, respectively.The necessary condition for a minimum of the objectivefunction (1) is of the form Va,JJ 0. Using the Euclideandistance as given in (17) we obtain

t9 ' =- 0 2Z Znu V k u (Wk,u,a - Xi,u,a)19Wk,u,a i,=1

ITIWk,u,a ZE 1 vik,u

z=-l

IT}

ZZViu42k,u Xi,u,ai=l

Wk,u,a - vITI n

i=l i,u i,k,u

which is also given with (6).

REFERENCES[1] D. J. Hand, H. Mannila, and P. Smyth, Principles ofData Mining. MIT

Press, 2001.[2] F. Hoppner, F. Klawoon, R. Kruse, and T. Runkler, Fuzzx Cluster

Analnsis. Chichester, England: John Wiley, 1999.[3] G. Cruciani, P. Crivori, P.-A. Carrupt, and B. Testa, "Molecular fields in

quantitative structure-permeation relationships: the VolSurf approach,"Journal of Molecular Structure, vol. 503, pp. 17-30, 2000.

[4] A. Schuffenhauer, V. J. Gillet, and P. Willett, "Similarity searching infiles of three-dimensional chemical structures: Analysis of the biosterdatabase using two-dimensional fingerprints and molecular field descrip-tors." Journal of Chemical Information and Computer Scienices, vol. 40,no. 2, pp. 295-307, 2000.

[5] A. Blum and T. Mitchell, "Combining labeled and unlabeled datawith co-training," in Proceedings of the eleventh annzual Coniference onzComnputational Learning Tlheoty (COLT'98). ACM Press, 1998, pp.92-100.

[6] K. Kailing, H.-P. Kriegel, A. Pryakhin, and M. Schubert, "Clusteringmulti-represented objects with noise." in PAKDD, 2004, pp. 394-403.

[7] D. E. Patterson and M. R. Berthold, "Clustering in parallel universes,"in Proceedings of the 2001 IEEE Conference in Systems, Man andCybernetics. IEEE Press, 2001.

[8] J. C. Bezdek, Pattern Recogniition with Fuzz.y Objective FunIction Algo-rithms. New York: Plenum Press, 1981.

[9] S. Bickel and T. Scheffer, "Multi-view clustering," in Proceedings ofthe Fourth IEEE Initernationzal Conference on Data Minintg (ICDM'04),2004. pp. 19-26.

[10] W. Pedrycz, "Collaborative fuzzy clustering," Pattern Recognition Let-ters, vol. 23, no. 14, pp. 1675-1686, 2002.

[11] J. Wang, H.-J. Zeng, Z. Chen, H. Lu, L. Tao, and W.-Y. Ma, "ReCoM:Reinforcement clustering of multi-type interrelated data objects." in InProceedings of the 26th aninual internationzal ACM SIGIR conferenceon7 research an2d development in information retrieval (SIGIR'03), 2003,pp. 274-281.

[12] R. R. Yager and D. P. Filev, "Approximate clustering via the mountainmethod," IEEE Trans. Systemns Man Cybernet., vol. 24, no. 8, pp. 1279-1284, August 1994.

[13] N. B. Venkateswarlu and P. S. V. S. K. Raju, "Fast ISODATA clusteringalgorithms," Pattern Recogniition, vol. 25, no. 3. pp. 335-342, 1992.

572

1-4 U)2 7-1

w XiVi,k,u du ( 'k,u,