Machine Learning Problems• Unsupervised Learning

– Clustering– Density estimation– Dimensionality Reduction

• Supervised Learning– Classification– Regression

Clustering• Clustering definition: Partition a given set of objects into M

groups (clusters) such that the objects of each group are ‘similar’ and ‘different’ from the objects of the other groups.

• A distance (or similarity) measure is required.• Unsupervised learning: no class labels• Clustering is NP-complete• Clustering Examples: documents, images, time series, image

segmentation, video analysis, gene clustering, motif discovery, web applications

• Big Issue: Number of Clusters estimation

• Difficult to evaluate solutions

Clustering• Cluster Assignments: hard vs (fuzzy/probabilistic)

• Clustering Methods− Hierarchical (agglomerative, divisive)− Density-based (non-parametric)− Parametric (k-means, mixture models etc)

• Clustering Problems– Data Vectors– Similarity Matrix

Agglomerative Clustering• The simplest approach and a good starting point• Starting from singleton clusters at each step we merge the

two most similar clusters• A similarity (or distance) measure between clusters is

needed• Output: dendrogram

• Drawback: merging decisions are permanent (cannot be corrected at a later stage)

Density-Based Clustering (eg DBSCAN)

• Identify ‘dense regions’ in the data space

• Merge neighboring dense regions

• Require a lot of points

• Complexity: O(n2)

Core

Border

Outlier

Eps = 1cm

MinPts = 5

(set empirically, how?)

Parametric methods• k-means (data vectors): O(n) (n: the number of objects to

be clustered)• k-medoids (similarity matrix): O(n2)

• Mixture models (data vectors): O(n)

• Spectral clustering (similarity matrix): O(n3)

• Kernel k-means (similarity matrix): O(n2)

• Affinity Propagation (similarity matrix): O(n2)

k-means• Partition a dataset X of N vectors xi into M subsets (clusters)

Ck such that intra-cluster variance is minimized.

• Intra-cluster variance: distance from the cluster prototype mk

• k-means: Prototype = cluster center• Finds local minima w.r.t. clustering error

– sum of intra-cluster variances

• Highly dependent on the initial positions (examples) of the centers mk

km (km.wmv)

k-medoids

• Similar to k-means • The represenative is the cluster medoid: the cluster object with

smallest average distance to the other cluster objects

• At each iteration the medoid is computed instead of centroid

• Increased complexity: O(n2)

• Medoid: more robust to outliers

• k-medoids can be used with similarity matrix

k-means (1) vs k-medoids (2)

10

Spectral Clustering (Ng & Jordan, NIPS2001)

Input: Similarity matrix between pairs of objects, number of clusters M

Example: a(x,y)=exp(-||x-y||2/σ2) (RBF kernel)

Spectral analysis of the similarity matrix: compute top M eigenvectorsand form matrix U

The i-th object corresponds to a vector in Rk : i-th row of U.

Rows are clustered in M clusters using k-means

1 2 3

1 0 01 0 00 1 00 1 00 0 10 0 1

U UU

11

Spectral Clustering

k-means spectral (RBF kernel,σ=1)

2 rings dataset

Spectral Clustering ↔ Graph cut

iji B,j C

(V,A) links(B,C)= a degree(B)=links(B,V)G

1 k 1 k

k kc c c c

V ,...,V V ,...,Vc=1 c=1c c

links(V ,V ) links(V ,V\V )RAss=max RCut=min

| V | | V |

1 k

kc c

V ,...,Vc=1 c

links(V ,V\V )NCut=min

degree(V )

Data graph

Vertices: objects

Edge weight: pairwise similarity

Clustering = Graph Partitioning

13

• Cluster Indicator vector zi=(0,0,…,0,1,0,…0)T for object i

• Indicator matrix Z=[z1,…,zn] (nxk, for k clusters), ZTZ=I

• Graph partitioning = trace maximization wrt Z:

• The relaxed problem: is solved optimally using the spectral algorithm to obtain Y

• k-means is applied on yij to obtain zij

TZmax trace(Z ), Z indicator matrix, function of Α

T Tijmax trace(Y Y), Y Y=I, yY R

Spectral Clustering ↔ Graph cut

Kernel-Based Clustering(non-linear cluster separation)

– Given a set of objects and the kernel matrix K=[Kij] containing the similarities between each pair of objects

– Goal: Partition the dataset into subsets (clusters) Ck such that intra-cluster similarity is maximized.

– Kernel trick: Data points are mapped from input space to a higher dimensional feature space through a transformation φ(x).

2( ) ( ), || ( ) ( ) || 2Tij i j i j ii jj ijK K K K x x x x

RBF kernel:

K(x,y)=exp(-||x-y||2 /σ2)

Kernel k-Means• Kernel k-means = k-means in feature space

– Minimizes the clustering error in feature space

• Differences from k-means– Cluster centers mk in feature space cannot be computed– Each cluster Ck is explicitly described by its data objects

– Computation of distances from centers in feature space:

• Finds local minima - Strong dependence on the initial partition

Spectral Relaxation of Kernel k-means

1 Dhillon, I.S., Guan, Y., Kulis, B., Weighted graph cuts without eigenvectors: A multilevel approach, IEEE TPAMI, 2007

Spectral methods can substitute kernel k-means and vice versa

Constant

Exemplar-Based Methods• Cluster data by identifying representative exemplars

– An exemplar is an actual dataset point, similar to a medoid– All data points are considered as possible exemplars– The number of clusters

is decided during learning (but a depends on a user-defined parameter)

• Methods– Convex Mixture Models– Affinity Propagation

Affinity Propagation (AP)(Frey et al., Science 2007)

• Clusters data by identifying representative exemplars– Exemplars are identified by transmitting messages between

data points

• Input to the algorithm– A similarity matrix where s(i,k) indicates how well data point xk

is suited to be an exemplar for data point xi.

– Self-similarities s(k,k) that control the number of identified clusters and a higher value means that xk is more likely to become an exemplar

• Self-similarities are independent of the other similarities• Higher values result in more clusters

Affinity Propagation• Clustering criterion:

– s(i,ci) is the similarity between the data point xi and its exemplar

– Minimized by passing messages between data points, called responsibilities and availabilities

• Responsibility r(i,k):– Sent from xi to candidate exemplar xk reflects the accumulated

evidence for how well suited xk is to serve as the exemplar of xi taking into account other potential exemplars for xi

Affinity Propagation• Availability a(i,k)

– Sent from candidate exemplar xk to xi reflects the accumulated evidence for how appropriate it would be for xi to choose xk as its exemplar, taking into account the support from other points that xk should be an exemplar

• The algorithm alternates between responsibility and availability calculation and

• The exemplars are the points with r(k,k)+a(k,k)>0– http://www.psi.toronto.edu/index.php?q=affinity%20propagation

Affinity Propagation

Incremental ClusteringBisecting k-means(Steinbach,Karypis & Kumar, SIGKDD 2000)

• Start with k=1 (m1= data average)• Assume a solution with k clusters

– Find the ‘best’ cluster split in two subclusters– Replace the cluster center with the two subcluster

centers– Run k-means with k+1 centers (optional)– k:=k+1

• Until M clusters have been added

• Split a cluster using several random trials• Each trial:

– Randomly initialize two centers from the cluster points– Run 2-means using the cluster points only

• Keep the split of the trial with the lowest clustering error

Global k-means(Likas, Vlassis & Verbeek, PR 2003)

• Incremental, deterministic clustering algorithm that runs k-Means several times

• Finds near-optimal solutions wrt clustering error

• Idea: a near-optimal solution for k clusters can be obtained by running k-means from an initial state

– the k-1 centers are initialized from a near-optimal solution of the (k-1)-clustering problem

– the k-th center is initialized at some data point xn (which?)

• Consider all possible initializations (one for each xn)

1 2 1( , ,..., , )k nm m m x

1 2 1( , ,..., )km m m

Global k-means

• In order to solve the M-clustering problem:– Solve the 1-clustering problem (trivial)– Solve the k-clustering problem using the solution of

the (k-1)-clustering problem • Execute k-Means N times, initialized as

at the n-th run (n=1,…,N).• Keep the solution corresponding to the run with the lowest

clustering error as the solution with k clusters

– k:=k+1, Repeat step 2 until k=M.

1 2 1( , ,..., , )k nm m m x

Best Initial m2

Best Initial m3

Best Initial m4

Best Initial m5

glkm (glkm.wmv)

100_clusters (100_clusters.wmv)

Fast Global k-Means

• How is the complexity reduced?– We select the initial state with the greatest reduction in clustering error in the first

iteration of k-means (reduction can be computed analytically)– k-means is executed only once from this state

• Restrict the set of candidate initial points (kd-tree, summarization)

*1 2 1( , ,..., , )k nm m m x

Global Kernel k-Means(Tzortzis & Likas, IEEE TNN 2009)

• In order to solve the M-clustering problem:1. Solve the 1-clustering problem with Kernel k-Means (trivial solution)2. Solve the k-clustering problem using the solution of the (k-1)-clustering

problema) Let denote the solution to the (k-1)-clustering problemb) Execute Kernel k-Means N times, initialized during the n-th run as

c) Keep the run with the lowest clustering error as the solution with k clusters

d) k := k+1

3. Repeat step 2 until k=M.

• The fast Global kernel k-means can be applied• Select representative data points using convex mixture models

1 2 1( , ,..., )kC C C

1 1( ,..., : { },...., , { }) l l n k k n n lC C C C C C x x x

Best Initial C3

Best Initial C4

Blue circles: optimal initialization of the cluster to be added

RBF kernel: K(x,y)=exp(-||x-y||2 /σ2)

Best Initial C2

Clustering Methods: SummaryUsually we assume that the number of clusters is given• k-means is still the most widely used method• Mixture models could be used when lots of data are available• Spectral clustering (or kernel k-means) the most popular when

similarity matrix is given

• Beware of the parameter initialization problem!• Ground truth absence makes evaluation difficult• How could we estimate the number of clusters?