A Unified View of Kernel k-means, Spectral Clustering and

Graph Cuts

Dhillon, Inderjit S., Yuqiang Guan and Brian Kulis

Outline

• (Kernel) kmean, weighted kernel kmean

• Spectral clustering algorithms

• The connect of kernel kmean and spectral clustering algorithms

• The Uniformed Problem and the ways to solve the problem

• Experiment results

K means and Kernel K means

Weighted Kernel k means

Matrix Form

Distance from ai to cluster c

Spectral Methods

• Represent the data by a graph– Each data points corresponds to a node on

the graph– The weight of the edge between two nodes

represent the similarity between the two corresponding data points

– The similarity can be a kernel function, such as the RBF kernel

• Use spectral theory to find the cut for the graph: Spectral Clustering

Spectral Methods

Spectral Methods

Similar in the cluster

Difference between clusters

Represented with Matrix

( , ) Tc c c clinks V V x Ax ( , \ ) T

c c c clinks V V V x Lx

| | Tc c cV x x ( ) T

c c cdegree V x Dx

L for Ncut

Ratio assoc

Ratio cut

Norm assoc

Weighted Graph CutWeighted association

Weighted cut

Conclusion

• Spectral Methods are special case of Kernel K means

Solve the unified problem

• A standard result in linear algebra states that if we relax the trace maximizations, such that Y is an arbitrary orthonormal matrix, then the optimal Y is of the form Vk Q, where Vk consists of the leading k eigenvectors of W1/2KW1/2 and Q is an arbitrary k × k orthogonal matrix.

• As these eigenvectors are not indicator vectors, we must then perform postprocessing on the eigenvectors to obtain a discrete clustering of the point

From Eigen Vector to Cluster Indicator

Normalized U with L2 norm equal to 1

2

1

The Other Way

• Using k means to solve the graph cut problem: (random start points+ EM, local optimal).

• To make sure k mean converge, the kernel matrix must be positive definite.

• This is not true for arbitrary kernel matrix

The effect of the regularizationai is in c

cai is not in

Experiment results

Results (ratio association)

Results (normalized association)

Image Segmentation

Thank you. Any Question?