A. Fender, N. Emad, S. Petiton, M. Naumov

SPECTRAL CLUSTERING OF LARGE NETWORKS

May 8th, 2017

2

Agenda

Introduction

Laplacian

Modularity

Conclusions

3

THE CLUSTERING PROBLEM

Example : detect relevant groups based on frequent co-purchasing on Amazon.com

Visualization: M. Bastian, S. Heymann, and M. Jacomy. “Gephi: An Open Source Software for exploring and manipulating networks” 2009

Data: V. Krebs. 2004

4

THE CLUSTERING PROBLEM

Pink Liberal

Yellow Neutral

Green Conservative

Data: V. Krebs. 2004

Visualization: M. Bastian, S. Heymann, and M. Jacomy. “Gephi: An Open Source Software for exploring and manipulating networks” 2009

5

CLUSTERING ALGORITHMS

• Spectral

Build a matrix, solve an eigenvalue problem, use eigenvectors for clustering

• Hierarchical / Agglomerative

Build a hierarchy (fine to coarse), partition coarse, propagate results back to fine level

• Local refinements

Switch one node at a time

A x x

fine coarse

6

Laplacian

7

RATIO CUT COST FUNCTION

Objective function:

𝐶𝑜𝑠𝑡 = 𝜕𝐸𝑖𝑉𝑖

𝑝

𝑖=1

where 𝜕𝐸𝑖 : # of edges cut and 𝑉𝑖 : # of nodes in i-th partition

A compromise between small edge-cut and balanced partitions 𝐶𝑜𝑠𝑡 =2

2+2

3=5

3

M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

8

1

1 1 1

1 1 1

1 1

1

GRAPH LAPLACIAN

L = D − A

D: degree matrix

A: adjacency matrix

1

3

3

2

1

1 -1

-1 3 -1 -1

-1 3 -1 -1

-1 -1 2

-1 1

= −

𝐿 𝐷 𝐴

1

2

3

4

5

M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

9

GRAPH LAPLACIAN

For a vector x with elements that are 0 or 1:

Number of edges cut

Number of elements

1

2

3

4

5

|𝜕𝐸1| =

1 -1

-1 3 -1 -1

-1 3 -1 -1

-1 -1 2

-1 1

L

1

1

x

1 1

xT

= 2

|𝑉1| = xT x = 2

V1 𝜕𝐸1

M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

10

MINIMIZATION PROBLEM

minxi xiTLxi

xiTxi

p

i=1

where xi ∈ 0,1n and xi ⊥ xj

min𝑉i 𝜕𝐸i𝑉i

p

i=1

1

2

3

4

5

V1 𝜕𝐸1

Next step Relax requirements on x, and let xi take real values

M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

11

K-MEANS POINTS CLUSTERING

Points Centroids

Lloyd’s Algorithm: • Select centroids • Compute distance of points to centroids • Assign points to the closest centroid • Recompute centroid position

p1

p2

pl

…

c1

ck

M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

12

EDGE CUT MINIMIZATION PIPELINE

Eigensolver Clustering Points

Clustering Graph

Preprocessing

1.0

1.0

1.0

1.0

1.0

𝑥1

-1.0

-0.3

0.3

0.0

1.0

𝑥2

1

1

𝑥1

1

1

1

𝑥2

Laplacian

1 -1

-1 3 -1 -1

-1 3 -1 -1

-1 -1 2

-1 1

13 Balanced cut minimization Ground truth

SPECTRAL EDGE CUT MINIMIZATION 80% hit rate

14

Modularity

15

Measures the difference between how well vertices are assigned into clusters for the current graph G = (V,E) versus a random graph R = (V,F).

G = (V,E) R = (V,F)

𝑄 =1

2𝜔 (𝑤𝑖𝑗

𝑗

−𝑣𝑖𝑣𝑗

2𝜔)

𝑖

𝛿𝑐 𝑖 𝑐(𝑗)

for some assignment c(.) into clusters.

MODULARITY FUNCTION

…

…

…

…

A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017

16

MODULARITY MATRIX

Let matrix

𝐵 = 𝐴 − 1

2𝜔𝑣𝑣𝑇

then modularity

Q =1

2𝜔 𝑇𝑟(𝑋𝑇𝐵𝑋)

where Tr(.) is the trace (sum of diagonal elements)

and X = [𝑥1, … , 𝑥𝑝] is such that 𝑥𝑖𝑘 = 1 𝑖𝑓 𝑐 𝑖 = 𝑘 .

1

2

3

4

5

1

1 1 1

1 1 1

1 1

1

𝐴 1

3

3

2

1

𝑣

A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017

17

MODULARITY MAXIMIZATION PIPELINE

Eigensolver Clustering Points

Clustering Graph

Preprocessing

-0.5

0.0

0.0

0.6

-0.5

𝑥4

-0.6

-0.4

0.4

0.0

0.6

𝑥5

1

1

𝑥1

1

1

1

𝑥2

Modularity

0 1

1 0 1 1

1 0 1 1

1 1 0

1 0

− 1

2𝜔𝑣𝑣𝑇

18

SPECTRAL MODULARITY MAXIMIZATION 84% hit rate

Spectral Modularity maximization Ground truth

19

PROFILING

The sparse matrix vector multiplication takes 90% of the time in the eigensolver

The eigensolver takes 90% of the time

20

MODULARITY VS. LAPLACIAN CLUSTERING

Nvidia Titan X (Pascal), Intel Core i7-3930K @3.2 GHz

Modularity

higher and steadier

modularity score

3x speedup over Laplacian

Laplacian

homogeneous cluster sizes

21

SPEEDUP AND QUALITY VS. AGGLOMERATIVE*

*: D. LaSalle and G. Karypis. “Multi-

threaded Modularity Based Graph

Clustering Using the Multilevel Paradigm”

Parallel Distrib. Comput., Vol. 76, pp. 66-

80, 2015.

Speedup

3x over agglomerative* scheme

Tradeoff

Speed vs. quality

0.8s on network with 100 million edges on a single Titan X GPU

Nvidia Titan X (Pascal), Intel Core i7-3930K @3.2 GHz

22

Spectral Clustering in CUDA Toolkit 9.0 release of nvGRAPH

23

CUDA TOOLKIT 9.0

nvGRAPH API

nvgraphStatus_t nvgraphSpectralClustering (

nvgraphHandle_t handle,

const nvgraphGraphDescr_t graph_descr,

const size_t weight_index,

const struct SpectralClusteringParameter *params,

int *clustering,

void *eig_vals,

void *eig_vects);

struct SpectralClusteringParameter {

int n_clusters;

int n_eig_vects;

nvgraphSpectralClusteringType_t alg

float evs_tolerance

int evs_max_iter;

float kmean_tolerance;

… };

24

Conclusions

25

SPECTRAL CLUSTERING

• Software Framework

• similar for both

• Laplacian - Minimum balanced cut[1]

• Probably the most common metric, with balancing involved in the cost function

• Requires careful choice of eigensolver

• Modularity maximization [2]

• Widely used in analysis of social networks

• Faster to compute

[1] M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016. [2] A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017.

26

Thank you H7129 - Accelerated Libraries Monday and Wednesday @ 4:00, Pod B