3

Activity: • Explain the fuzzy c-means clustering technique • Try to cluster data points into fuzzy clusters using the FCMC technique

Given a set of data, clustering techniques partition the data into several groups such that the degree of association is strong within one group and weak between data in different groups. Classical crisp clustering techniques result in crisp partitions where each data point can belong to only one cluster. This situation is illustrated in Fig. 3-1-1 where six data points are clustered into two clusters. In this example the number of dimensions of data is 2. (i.e. the so called feature space is two-dimensional.)

Fig. 3-1-1. A simple example for crisp clustering

Fuzzy clustering by contrast allows data points to belong to more than one group. The resulting partition is therefore a fuzzy partition. Each cluster is associated with a membership function that expresses the degree to which individual data points belong to the cluster. Among all fuzzy clustering methods, Fuzzy c-Means Clustering (FCMC) remains predominant in the literature [2] due to its successful application both in the academia as well as industry. Activity:

• Observe that the number of clusters should be given in advance • Observe the iterativity in the FCMC algorithm

Fuzzy c-Means Clustering performs clustering by iteratively searching for a set of fuzzy clusters and the associated cluster centres that represent the structure of the data as best as possible. The algorithm relies on the user to specify the number of clusters present in the set of data to be clustered. Given a number of clusters c, FCMC partitions the data X = {x1,x2,…,xn} into c fuzzy clusters by minimising the within group sum of squared error objective function as follows:

4

∞≤≤

−= ∑∑= =

m

vxUVUJn

k

c

iik

mikm

1

,||||)(),(1 1

2

(1)

where Jm(U,V) is the sum of squared error for the set of fuzzy clusters represented by the membership matrix U, and the associated set of cluster centres V. ||.|| is some inner product induced norm. In the formula, ||xk – vi||2 represents the distance between the data xk and the cluster centre vi. The squared error is used as a performance index that measures the weighted sum of distances between cluster centres and elements in the corresponding fuzzy clusters. The number m governs the influence of membership grades in the performance index. The partition becomes fuzzier with increasing m and it has been shown that the FCMC algorithm converges for any m ∈ (1,∞). The necessary conditions for Eq. (1) to reach its minimum are

kivxvxU

c

j

m

jk

ikik ∀∀

−−

=

−

=

−

∑ ,||||||||

1

1

)1/(2

(2)

and

∑

∑

=

== n

k

mik

n

kk

mik

i

U

xUv

1

1

)(

)(

(3)

In each iteration of the FCMC algorithm, matrix U is computed using Eq. (2) and the associated cluster centres are computed as Eq. (3). This is followed by computing the square error in Eq. (1). The algorithm stops when either the error is below a certain tolerance value or its improvement over the previous iteration is below a certain threshold. The clustering process is displayed in Fig. 3-1-2 (initial state) and Fig. 3-1-3 (final state) using three clusters.

5

Fig. 3-1-2. Clustering process (initial state)

Fig. 3-1-3. Clustering process (final state)

Activity:

• See the different types of clustering algorithms, note their advantages Over the years, many extensions and variations of FCMC have been proposed [5]. In general, the use of different distance functions in Eq. (1) or a slight modification of the objective function leads to clustering algorithms capable of detecting different types of clusters. Using the Euclidean distance, FCMC is capable of detecting approximately similar-sized spherical clusters. Gustafson and Kessel in [6] proposed the use of a transformed Mahalanobis distance, allowing the resulting GK clustering algorithm to detect cylinder-shaped normal clusters of approximately the same size. The technique was analysed and compared to other clustering techniques in [7]. The distance used is Eq. (4). In the equation, Ci is the covariance matrix for

6

the i-th cluster and Ci-1 denotes the covariance matrix inverse, d is the number of dimensions

and ρi = 1, is a constant.

)vx(C)vx(Cd ik1

iT

ikd/1

id/1

i2ik −−ρ= −

(4)

Fuzzy c-Elliptotypes (FCE) was designed to find linear (flat) clusters. The algorithm uses the distance in Eq. (5). Here, dEik

2 denotes the Euclidean distance between the cluster i and data point k; {ei 1, ei2, …, eir} are the eigenvectors (arranged in descending order of the corresponding eigenvalues) of the covariance matrix of clusters i; and r, with 0 ≤ r ≤ (d-1) represents the number of dimensions in which the flatness extends in the d-dimensional space. By the choice of α, the cluster shape can be changed from point-shape (α = 0) via elliptic shapes (α ∈ [0, 1]) to straight lines (α = 1).

[ ]∑=

−−−=

α−+α=r

1k

2ikij

2

ij2Lij

2Eik

2Lik

2ik

e)mx(mxd

d)1(dd

(5)

The Fuzzy c-Shells (FCS) Algorithm is designed to detect circles. The distance used involves a circle radius r, is defined as Eq. (6).

2ik

2ik )rvx(d −−=

(6)

Rectangular clusters can be detected by the Fuzzy c-Rectangular Shells (FCRS) algorithm using Eq. (7). Here, ns, rs are as shown in Fig. 3-1-4.

{ }{ } 22modss

Tik

2ik )3,2,1,0s|rn)vx((mind ∈+−=

(7)

Fig. 3-1-4. FCRS parameters

Table 1. summarises some of the well known cluster algorithms that have not been discussed and the corresponding types of clusters detectable by the algorithms [8].

7

The review of the extensions and variations of FCMC suggests that no one fuzzy clustering algorithm is universally applicable. The effectiveness of the algorithm depends heavily on the data at hand. When the cluster’s attributes are known (e.g. size and shape), the most suitable technique can be chosen accordingly. If such prior knowledge is not available (as is often the case), assumptions have to be made. Clustering Algorithm Types of clusters detectable by the

algorithm Gath Geva clustering (GG) Ellipsoidal clusters with varying size Adaptive Fuzzy c-Varieties (AFC) Line segments in 2D data Fuzzy c-Spherical Shells algorithm (FCSS) Clusters of circle shape Fuzzy c-Rings algorithm (FCR) Clusters of circle shape Fuzzy c-Quadratic Shells algorithm (FCQS) Ellipsoids

Table 1. Clustering algorithm capable of detecting different types of clusters