1

Chapter 8: Clustering

2

Searching for groups Clustering is unsupervised or

undirected. Unlike classification, in clustering, no

pre-classified data. Search for groups or clusters of data

points (records) that are similar to one another.

Similar points may mean: similar customers, products, that will behave in similar ways.

3

Group similar points together

Group points into classes using some distance measures. Within-cluster distance, and between

cluster distance Applications:

As a stand-alone tool to get insight into data distribution

As a preprocessing step for other algorithms

4

An Illustration

5

Examples of Clustering Applications

Marketing: Help marketers discover distinct groups in their customer bases, and then use this knowledge to develop targeted marketing programs

Insurance: Identifying groups of motor insurance policy holders with some interesting characteristics.

City-planning: Identifying groups of houses according to their house type, value, and geographical location

6

Concepts of Clustering

Clusters Different ways of

representing clusters Division with

boundaries Spheres Probabilistic Dendrograms …

1 2 3

I1

I2

…

In

0.5 0.2 0.3

7

Clustering Clustering quality

Inter-clusters distance maximized Intra-clusters distance minimized

The quality of a clustering result depends on both the similarity measure used by the method and its application.

The quality of a clustering method is also measured by its ability to discover some or all of the hidden patterns

Clustering vs. classification Which one is more difficult? Why? There are a huge number of clustering techniques.

8

Dissimilarity/Distance Measure

Dissimilarity/Similarity metric: Similarity is expressed in terms of a distance function, which is typically metric: d (i, j)

The definitions of distance functions are usually very different for interval-scaled, boolean, categorical, ordinal and ratio variables.

Weights should be associated with different variables based on applications and data semantics.

It is hard to define “similar enough” or “good enough”. The answer is typically highly subjective.

9

Types of data in clustering analysis

Interval-scaled variables

Binary variables

Nominal, ordinal, and ratio variables

Variables of mixed types

10

Interval-valued variables Continuous measurements in a roughly

linear scale, e.g., weight, height, temperature, etc

Standardize data (depending on applications) Calculate the mean absolute deviation:

where Calculate the standardized measurement (z-

score)

.)...21

1nffff

xx(xn m

|)|...|||(|121 fnffffff

mxmxmxns

f

fifif s

mx z

11

Similarity Between Objects Distance: Measure the similarity or

dissimilarity between two data objects Some popular ones include: Minkowski

distance:

where (xi1, xi2, …, xip) and (xj1, xj2, …, xjp) are

two p-dimensional data objects, and q is a positive integer

If q = 1, d is Manhattan distance

qq

pp

qq

jx

ix

jx

ix

jx

ixjid )||...|||(|),(

2211

||...||||),(2211 pp jxixjxixjxixjid

12

Similarity Between Objects (Cont.)

If q = 2, d is Euclidean distance:

Properties d(i,j) 0 d(i,i) = 0 d(i,j) = d(j,i) d(i,j) d(i,k) + d(k,j)

Also, one can use weighted distance, and many other similarity/distance measures.

)||...|||(|),( 22

22

2

11 pp jx

ix

jx

ix

jx

ixjid

13

Binary Variables A contingency table for binary data

Simple matching coefficient (invariant, if

the binary variable is symmetric):

Jaccard coefficient (noninvariant if the

binary variable is asymmetric):

dcbacb jid

),(

pdbcasum

dcdc

baba

sum

0

1

01

cbacb jid

),(

Object i

Object j

14

Dissimilarity of Binary Variables

Example

gender is a symmetric attribute (not used below) the remaining attributes are asymmetric

attributes let the values Y and P be set to 1, and the value

N be set to 0

Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4

Jack M Y N P N N NMary F Y N P N P NJim M Y P N N N N

75.0211

21),(

67.0111

11),(

33.0102

10),(

maryjimd

jimjackd

maryjackd

15

Nominal Variables A generalization of the binary variable in that

it can take more than 2 states, e.g., red, yellow, blue, green, etc

Method 1: Simple matching m: # of matches, p: total # of variables

Method 2: use a large number of binary variables creating a new binary variable for each of the M

nominal states

pmpjid ),(

16

Ordinal Variables

An ordinal variable can be discrete or continuous Order is important, e.g., rank Can be treated like interval-scaled (f is a variable)

replace xif by their ranks

map the range of each variable onto [0, 1] by replacing i-th object in the f-th variable by

compute the dissimilarity using methods for interval-scaled variables

11

f

ifif M

rz

},...,1{fif

Mr

17

Ratio-Scaled Variables Ratio-scaled variable: a measurement on a

nonlinear scale, approximately at exponential scale, such as AeBt or Ae-Bt, e.g., growth of a bacteria population.

Methods: treat them like interval-scaled variables—not a good idea!

(why?—the scale can be distorted) apply logarithmic transformation

yif = log(xif) treat them as continuous ordinal data and then treat their

ranks as interval-scaled

18

Variables of Mixed Types A database may contain all six types of

variables symmetric binary, asymmetric binary, nominal,

ordinal, interval and ratio One may use a weighted formula to combine

their effects

f is binary or nominal:dij

(f) = 0 if xif = xjf , or dij(f) = 1 o.w.

f is interval-based: use the normalized distance f is ordinal or ratio-scaled

compute ranks rif and and treat zif as interval-scaled

)(1

)()(1),(

fij

pf

fij

fij

pf

djid

1

1

f

if

Mrz

if

19

Major Clustering Techniques

Partitioning algorithms: Construct various partitions and then evaluate them by some criterion

Hierarchy algorithms: Create a hierarchical decomposition of the set of data (or objects) using some criterion

Density-based: based on connectivity and density functions

Model-based: A model is hypothesized for each of the clusters and the idea is to find the best fit of the model to each other.

20

Partitioning Algorithms: Basic Concept

Partitioning method: Construct a partition of a database D of n objects into a set of k clusters

Given a k, find a partition of k clusters that optimizes the chosen partitioning criterion Global optimal: exhaustively enumerate all partitions Heuristic methods: k-means and k-medoids algorithms k-means : Each cluster is represented by the center of

the cluster k-medoids or PAM (Partition around medoids): Each

cluster is represented by one of the objects in the cluster

21

The K-Means Clustering Given k, the k-means algorithm is as

follows: 1) Choose k cluster centers to coincide with k

randomly-chosen points2) Assign each data point to the closest cluster

center 3) Recompute the cluster centers using the

current cluster memberships.4) If a convergence criterion is not met, go to

2).Typical convergence criteria are: no (or minimal)

reassignment of data points to new cluster centers, or minimal decrease in squared error.

2

1

||

k

iCp i

impE

p is a point and mi is the mean of cluster Ci

22

Example For simplicity, 1 dimensional data and k=2. data: 1, 2, 5, 6,7 K-means:

Randomly select 5 and 6 as initial centroids; => Two clusters {1,2,5} and {6,7};

meanC1=8/3, meanC2=6.5 => {1,2}, {5,6,7}; meanC1=1.5, meanC2=6 => no change. Aggregate dissimilarity = 0.5^2 + 0.5^2 + 1^2

+ 1^2 = 2.5

23

Comments on K-Means Strength: efficient: O(tkn), where n is # data points, k

is # clusters, and t is # iterations. Normally, k, t << n. Comment: Often terminates at a local optimum. The

global optimum may be found using techniques such as: deterministic annealing and genetic algorithms

Weakness Applicable only when mean is defined, difficult for categorical

data Need to specify k, the number of clusters, in advance Sensitive to noisy data and outliers Not suitable to discover clusters with non-convex shapes Sensitive to initial seeds

24

Variations of the K-Means Method

A few variants of the k-means which differ in Selection of the initial k seeds Dissimilarity measures Strategies to calculate cluster means

Handling categorical data: k-modes Replacing means of clusters with modes Using new dissimilarity measures to deal with

categorical objects Using a frequency based method to update

modes of clusters

25

k-Medoids clustering method

k-Means algorithm is sensitive to outliers Since an object with an extremely large value may

substantially distort the distribution of the data. Medoid – the most centrally located point in a

cluster, as a representative point of the cluster.

An example

In contrast, a centroid is not necessarily inside a cluster.

Initial Medoids

26

Partition Around Medoids PAM:

1. Given k2. Randomly pick k instances as initial medoids3. Assign each data point to the nearest

medoid x4. Calculate the objective function

the sum of dissimilarities of all points to their nearest medoids. (squared-error criterion)

5. Randomly select an point y6. Swap x by y if the swap reduces the

objective function7. Repeat (3-6) until no change

27

Comments on PAM Pam is more robust than k-means in

the presence of noise and outliers because a medoid is less influenced by outliers or other extreme values than a mean (why?)

Pam works well for small data sets but does not scale well for large data sets. O(k(n-k)2 ) for each change

where n is # of data, k is # of clusters

Outlier (100 unit away)

28

CLARA: Clustering Large Applications CLARA: Built in statistical analysis packages,

such as S+ It draws multiple samples of the data set,

applies PAM on each sample, and gives the best clustering as the output

Strength: deals with larger data sets than PAM Weakness:

Efficiency depends on the sample size A good clustering based on samples will not

necessarily represent a good clustering of the whole data set if the sample is biased

There are other scale-up methods e.g., CLARANS

29

Hierarchical Clustering Use distance matrix for clustering. This

method does not require the number of clusters k as an input, but needs a termination condition

Step 0 Step 1 Step 2 Step 3 Step 4

b

d

c

e

a a b

d e

c d e

a b c d e

Step 4 Step 3 Step 2 Step 1 Step 0

agglomerative

divisive

30

Agglomerative Clustering

At the beginning, each data point forms a cluster (also called a node). Merge nodes/clusters that have the least dissimilarity.Go on mergingEventually all nodes belong to the same cluster

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

31

A Dendrogram Shows How the Clusters are Merged Hierarchically

Decompose data objects into a several levels of nested partitioning (tree of clusters), called a dendrogram.

A clustering of the data objects is obtained by cutting the dendrogram at the desired level, then each connected component forms a cluster.

32

Divisive Clustering

Inverse order of agglomerative clustering

Eventually each node forms a cluster on its own

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

33

More on Hierarchical Methods

Major weakness of agglomerative clustering methods do not scale well: time complexity at least O(n2),

where n is the total number of objects can never undo what was done previously

Integration of hierarchical with distance-based clustering to scale-up these clustering methods BIRCH (1996): uses CF-tree and incrementally

adjusts the quality of sub-clusters CURE (1998): selects well-scattered points from the

cluster and then shrinks them towards the center of the cluster by a specified fraction

34

Summary Cluster analysis groups objects based on

their similarity and has wide applications Measure of similarity can be computed for

various types of data Clustering algorithms can be categorized

into partitioning methods, hierarchical methods, density-based methods, etc

Clustering can also be used for outlier detection which are useful for fraud detection

What is the best clustering algorithm?