3
THE CLUSTERING PROBLEM
Example : detect relevant groups based on frequent co-purchasing on Amazon.com
Visualization: M. Bastian, S. Heymann, and M. Jacomy. “Gephi: An Open Source Software for exploring and manipulating networks” 2009
Data: V. Krebs. 2004
4
THE CLUSTERING PROBLEM
Pink Liberal
Yellow Neutral
Green Conservative
Data: V. Krebs. 2004
Visualization: M. Bastian, S. Heymann, and M. Jacomy. “Gephi: An Open Source Software for exploring and manipulating networks” 2009
5
CLUSTERING ALGORITHMS
• Spectral
Build a matrix, solve an eigenvalue problem, use eigenvectors for clustering
• Hierarchical / Agglomerative
Build a hierarchy (fine to coarse), partition coarse, propagate results back to fine level
• Local refinements
Switch one node at a time
A x x
fine coarse
7
RATIO CUT COST FUNCTION
Objective function:
𝐶𝑜𝑠𝑡 = 𝜕𝐸𝑖𝑉𝑖
𝑝
𝑖=1
where 𝜕𝐸𝑖 : # of edges cut and 𝑉𝑖 : # of nodes in i-th partition
A compromise between small edge-cut and balanced partitions 𝐶𝑜𝑠𝑡 =2
2+2
3=5
3
M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.
8
1
1 1 1
1 1 1
1 1
1
GRAPH LAPLACIAN
L = D − A
D: degree matrix
A: adjacency matrix
1
3
3
2
1
1 -1
-1 3 -1 -1
-1 3 -1 -1
-1 -1 2
-1 1
= −
𝐿 𝐷 𝐴
1
2
3
4
5
M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.
9
GRAPH LAPLACIAN
For a vector x with elements that are 0 or 1:
Number of edges cut
Number of elements
1
2
3
4
5
|𝜕𝐸1| =
1 -1
-1 3 -1 -1
-1 3 -1 -1
-1 -1 2
-1 1
L
1
1
x
1 1
xT
= 2
|𝑉1| = xT x = 2
V1 𝜕𝐸1
M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.
10
MINIMIZATION PROBLEM
minxi xiTLxi
xiTxi
p
i=1
where xi ∈ 0,1n and xi ⊥ xj
min𝑉i 𝜕𝐸i𝑉i
p
i=1
1
2
3
4
5
V1 𝜕𝐸1
Next step Relax requirements on x, and let xi take real values
M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.
11
K-MEANS POINTS CLUSTERING
Points Centroids
Lloyd’s Algorithm: • Select centroids • Compute distance of points to centroids • Assign points to the closest centroid • Recompute centroid position
p1
p2
pl
…
c1
ck
M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.
12
EDGE CUT MINIMIZATION PIPELINE
Eigensolver Clustering Points
Clustering Graph
Preprocessing
1.0
1.0
1.0
1.0
1.0
𝑥1
-1.0
-0.3
0.3
0.0
1.0
𝑥2
1
1
𝑥1
1
1
1
𝑥2
Laplacian
1 -1
-1 3 -1 -1
-1 3 -1 -1
-1 -1 2
-1 1
15
Measures the difference between how well vertices are assigned into clusters for the current graph G = (V,E) versus a random graph R = (V,F).
G = (V,E) R = (V,F)
𝑄 =1
2𝜔 (𝑤𝑖𝑗
𝑗
−𝑣𝑖𝑣𝑗
2𝜔)
𝑖
𝛿𝑐 𝑖 𝑐(𝑗)
for some assignment c(.) into clusters.
MODULARITY FUNCTION
…
…
…
…
A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017
16
MODULARITY MATRIX
Let matrix
𝐵 = 𝐴 − 1
2𝜔𝑣𝑣𝑇
then modularity
Q =1
2𝜔 𝑇𝑟(𝑋𝑇𝐵𝑋)
where Tr(.) is the trace (sum of diagonal elements)
and X = [𝑥1, … , 𝑥𝑝] is such that 𝑥𝑖𝑘 = 1 𝑖𝑓 𝑐 𝑖 = 𝑘 .
1
2
3
4
5
1
1 1 1
1 1 1
1 1
1
𝐴 1
3
3
2
1
𝑣
A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017
17
MODULARITY MAXIMIZATION PIPELINE
Eigensolver Clustering Points
Clustering Graph
Preprocessing
-0.5
0.0
0.0
0.6
-0.5
𝑥4
-0.6
-0.4
0.4
0.0
0.6
𝑥5
1
1
𝑥1
1
1
1
𝑥2
Modularity
0 1
1 0 1 1
1 0 1 1
1 1 0
1 0
− 1
2𝜔𝑣𝑣𝑇
19
PROFILING
The sparse matrix vector multiplication takes 90% of the time in the eigensolver
The eigensolver takes 90% of the time
20
MODULARITY VS. LAPLACIAN CLUSTERING
Nvidia Titan X (Pascal), Intel Core i7-3930K @3.2 GHz
Modularity
higher and steadier
modularity score
3x speedup over Laplacian
Laplacian
homogeneous cluster sizes
21
SPEEDUP AND QUALITY VS. AGGLOMERATIVE*
*: D. LaSalle and G. Karypis. “Multi-
threaded Modularity Based Graph
Clustering Using the Multilevel Paradigm”
Parallel Distrib. Comput., Vol. 76, pp. 66-
80, 2015.
Speedup
3x over agglomerative* scheme
Tradeoff
Speed vs. quality
0.8s on network with 100 million edges on a single Titan X GPU
Nvidia Titan X (Pascal), Intel Core i7-3930K @3.2 GHz
23
CUDA TOOLKIT 9.0
nvGRAPH API
nvgraphStatus_t nvgraphSpectralClustering (
nvgraphHandle_t handle,
const nvgraphGraphDescr_t graph_descr,
const size_t weight_index,
const struct SpectralClusteringParameter *params,
int *clustering,
void *eig_vals,
void *eig_vects);
struct SpectralClusteringParameter {
int n_clusters;
int n_eig_vects;
nvgraphSpectralClusteringType_t alg
float evs_tolerance
int evs_max_iter;
float kmean_tolerance;
… };
25
SPECTRAL CLUSTERING
• Software Framework
• similar for both
• Laplacian - Minimum balanced cut[1]
• Probably the most common metric, with balancing involved in the cost function
• Requires careful choice of eigensolver
• Modularity maximization [2]
• Widely used in analysis of social networks
• Faster to compute
[1] M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016. [2] A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017.