Upload
others
View
18
Download
0
Embed Size (px)
Citation preview
Spectral Clustering
Tapesh Yadav
Final PresentationE0 270: Machine Learning
Instructor: Ambedkar Dukkipati
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 1
/ 36
Outline
1 similarity graphs and Clustering
2 Graph Laplacians
3 Spectral Clustering Algorithms
4 Various Interpretations
5 Reproducing Results
6 Experiments
7 Clustering on spiral dataset
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 2
/ 36
Graph notation and similarity graphs
Clustering problem can be reformulated as follows:
Problem
Find a partition of the graph such that the edges between different groupshave a very low weight (which means that points in different clusters aredissimilar from each other) and the edges within a group have high weight(which means that points within the same cluster are similar to eachother).
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 3
/ 36
Different similarity graphs
ε neighbourhood Graph
Connect all points whose pairwise distances are smaller than ε.
Unweighted
k-nearest neighbor graphs (knn)
Connect vertex vi with vertex vj if vj is amongst k nearest neighboursof vi .
Use mutual k nearest neighbour method or k nearest neighbour.to get undirected graph.
The fully connected graph:
Here we simply connect all points with positive similarity with eachother, and we weight all edges by sij .
Gaussian similarity function, s(xi , xj) = exp(||xi−xj ||2
2σ2
).
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 4
/ 36
Graph Laplacians
The main tools for spectral clustering are graph Laplacian matrices.
Unormalized graph laplacian, L = D −W .
Normalized graph laplacians:
Lsym = D−1/2LD−1/2
Lrw = D−1L
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 5
/ 36
Unnormalized spectral clustering
Input: Similarity matrix S ∈ Rnxn, number k of clusters to construct.
Construct a similarity graph by one of the ways mentioned previously.Let W be its weighted adjacency matrix.
Compute the unnormalized Laplacian L.
Compute the first k eigenvectors u1, ..., uk of L.
Let U ∈ Rnxk be the matrix containing the vectors u1, ..., uk ascolumns.
For i = 1, ..., n, let yi ∈ Rk be the vector corresponding to the i-throw of U.
Cluster the points (yi )i=1,...,n in Rk with the k-means algorithm intoclusters C1, ...,Ck .
Output: Clusters A1, ...,Ak with Ai = {yj ∈ Ci}.
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 6
/ 36
Normalized spectral clustering according to Shi and Malik(2000)
Input: Similarity matrix S ∈ Rnxn, number k of clusters to construct.
Construct a similarity graph by one of the ways mentioned previously.Let W be its weighted adjacency matrix.
Compute the normalized Laplacian Lrw .
Compute the first k generalized eigenvectors u1, ..., uk of thegeneralized eigenproblem Lu = λDu.
Let U ∈ Rnxk be the matrix containing the vectors u1, ..., uk ascolumns.
For i = 1, ..., n, let yi ∈ Rk be the vector corresponding to the i-throw of U.
Cluster the points (yi )i=1,...,n in Rk with the k-means algorithm intoclusters C1, ...,Ck .
Output: Clusters A1, ...,Ak with Ai = {yj ∈ Ci}.
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 7
/ 36
Normalized spectral clustering according to Ng, Jordan,and Weiss (2002)
Input: Similarity matrix S ∈ Rnxn, number k of clusters to construct.
Construct a similarity graph by one of the ways mentioned previously.Let W be its weighted adjacency matrix.
Compute the unnormalized Laplacian L.
Compute the first k eigenvectors u1, ..., uk of Lsym.
Let U ∈ Rnxk be matrix containing the vectors u1, ..., uk as columns.
Form the matrix T ∈ Rnxk from U by normalizing the rows to norm 1.
For i = 1, ..., n, let yi ∈ Rk be the vector corresponding to the i-throw of T.
Cluster the points (yi )i=1,...,n in Rk with the k-means algorithm intoclusters C1, ...,Ck .
Output: Clusters A1, ...,Ak with Ai = {yj ∈ Ci}.
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 8
/ 36
Cut, RatioCut and Ncut
for a partition A,B of V, define
cut(A,B) =∑
i∈A,j∈Bwij
We mention here three objective functions, minimizing which broadlyresult in clusters having minimized intercluster distance.
cut(A1,A2, . . . ,Ak) =∑k
i=1 cut(Ai , Ai )
RatioCut(A1, ...,Ak) =∑k
i=1cut(Ai ,Ai )|Ai | (Explains unnormalised
spectral clustering).
Ncut(A1, ...,Ak) =∑k
i=1cut(Ai ,Ai )vol(Ai )
(Explains normalized spectral
clustering).
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 9
/ 36
Other point of views
Spectral clustering (Ncut) can be explained using random walk ongraphs (P = D−1W =⇒ Lrw = I − P).
Perturbation theory also gives a beautiful explanation.
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 10
/ 36
Figure 1 from paper
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 11
/ 36
Reproducing Figure 1 with 196 data points
Figure: normalized, knn
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 12
/ 36
Reproducing Figure 1 with 196 data points
Figure: unnormalized, knn
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 13
/ 36
Reproducing Figure 1 with 196 data points
Figure: normalized, full graph
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 14
/ 36
Reproducing Figure 1 with 196 data points
Figure: unnormalized, full graph
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 15
/ 36
Figure 2 from paper
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 16
/ 36
Reproducing Figure 2
Figure: Cockroach graph
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 17
/ 36
Reproducing Figure 2
Figure: kmeans clustering
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 18
/ 36
Reproducing Figure 2
Figure: sign based clustering
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 19
/ 36
Figure 3 from Paper
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 20
/ 36
Reproducing Figure 3
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 21
/ 36
Figure 4 from Paper
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 22
/ 36
Reproducing Figure 4
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 23
/ 36
Figure 5 from Paper
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 24
/ 36
Reproducing Figure 5
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 25
/ 36
Using different metrics for kmeans
Figure: squared norm metric
Figure: cityblock metric
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 26
/ 36
Higher Eigenvectors of Laplacian
Figure: Higher Eigenvectors (5th and 6th) for unnormalized knn
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 27
/ 36
8 clusters: normalized, full graph
Figure: Data histogram and first 4 eigenvectors
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 28
/ 36
8 clusters: normalized, full graph
Figure: Clusters and last four eigenvectors
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 29
/ 36
High variance input data
Figure: Data histogram and first 4 eigenvectors
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 30
/ 36
High variance input data
Figure: Clusters and last two eigenvectors
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 31
/ 36
Dirac delta like behaviour of bigger eigenvectors for fullgraph, unnormalised clustering
Figure: Data histogram and first 4 eigenvectors
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 32
/ 36
Dirac delta like behaviour of bigger eigenvectors for fullgraph, unnormalised clustering
Figure: Clusters and last two eigenvectors
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 33
/ 36
Clustering on spiral dataset
Figure: Clustering using unnormalized, mutual knn
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 34
/ 36
Clustering on spiral dataset
Figure: Clustering using unnormalized, epsilon neighbour
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 35
/ 36
References
Ulrike von Luxburg, A Tutorial on Spectral Clustering,http://arxiv.org/abs/0711.0189, 2007
Tapesh Yadav (Indian Institute of Science) Spectral ClusteringFinal Presentation E0 270: Machine Learning Instructor: Ambedkar Dukkipati 36
/ 36