Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Semidefinite Relaxation for the CliquePartitioning and Clustering Problems
Brendan Ames1 Stephen Vavasis1
1Department of Combinatorics and OptimizationUniversity of Waterloo
MS144: Semidefinite Approaches to Combinatorial Problems -Part I of III
ICIAM 2011, Vancouver, July 18-22
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Outline
1 Clusters, cliques, and low-rank matrices
2 The maximum node KDC problem
3 The weighted KDC problem
4 Numerical results
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Average vs Planted Case analysis
• Classical idea: study the behaviour of problems for randomprogram inputs.
• Algorithms or heuristics that perform poorly in general maystill work for most program instances (eg. simplex).
• For many problems, a truly random program instance is not agood model for a generic instance.
• Matrix completion: matrix to be recovered may be necessarilyof low-rank (eg. partially observed EDM in low-dimensionalspace, matrix of customer preferences in recommendationsystems).
• Image processing: natural images tend to have large regions ofsimilar pixels separated by sharp edges.
• A better model for a generic instance for these problems is theplanted case: generate a program instance where the desiredhidden structure has been obscured by random noise.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Rank minimization
• Affine rank minimization problem (RMP): find matrix withminimum rank satisfying linear constraints:
minrank(X ) : A(X ) = b
(here A : Rm×n → Rp linear, b ∈ Rp).
• NP-hard but can be solved in the case that A is random anda low-rank solution exists by relaxing to a convex program.
• Replace rank(X ) with nuclear norm‖X‖∗ = σ1(X ) + · · ·+ σm(X ). (NNM)
• Fazel (2002), Recht-Fazel-Parrilo (2007), Candes-Recht(2008).
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
The clustering problem
• Clustering: Want to partition a given data set so that items ineach cluster are similar to each other and items not in thesame cluster are dissimilar.
• Intractible in general.
• If data is actually clustered, can we identify the clustersefficiently?
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Clustering as graph partitioning
• Can model data as graph G = (V ,E ).
• V is the set of items in the data.
• Similarity is indicated by E :• Either by adjacency (ij ∈ E iff i , j are similar), or• By assigning edge-weights to each edge of complete graph.
• A set of nodes C ⊆ V is a clique of G if ij ∈ E for all i , j ∈ C .
• Every clique of G (with char. vector v) defines a rank-oneblock of AG + I by vvT .
• Optimal partitioning of the graph into cliques gives an optimalclustering of the data.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
The k-disjoint-clique problem
• Given graph G = (V ,E ) and integer k ∈ [1, |V |], ak-disjoint-clique-subgraph is a subgraph of G composed of kdisjoint cliques.
• We consider the problem of identifying the largestk-disjoint-clique subgraph in a given graph G .
• binary case: maximize number of nodes.
• weighted case: maximize normalized edge-weights.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
The maximum node KDC problem
The maximum node k-disjoint-clique problem (KDC):
Find a k-disjoint-clique subgraph containing the maximum numberof nodes of input graph G .
Special case: k = 1 The maximum clique problem
[A-V 09] if input graph contains a sufficiently large clique(Ω(√N)), can solve the maximum clique problem exactly by
formulating as rank minimization and relaxing to nuclear normminimization (even though A does not satisfy RIP or incoherence).
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
The maximum node KDC problem
The maximum node k-disjoint-clique problem (KDC):
Find a k-disjoint-clique subgraph containing the maximum numberof nodes of input graph G .
Special case: k = 1 The maximum clique problem
[A-V 09] if input graph contains a sufficiently large clique(Ω(√N)), can solve the maximum clique problem exactly by
formulating as rank minimization and relaxing to nuclear normminimization (even though A does not satisfy RIP or incoherence).
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
What if we want to find many cliques?
• In general, can’t formulate k-disjoint-clique as rankminimization.
• k may be very large, don’t want to recover a low-rank matrix.
• Can’t generate series of cuts using the maximum cliquebecause not in the planted case.
• Solution: Relax rank constraints with combination of nuclearnorm constraint and semidefinite constraint.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Relaxation as SDP
• We relax KDC as the SDP
max∑∑
Xij
s.t. Xe ≤ e,(∗) Xij = 0, ∀(i , j) /∈ E , i 6= j
tr(X ) = k ,X 0.
• Xe ≤ e⇒ ‖X‖∗ ≤ rank(X ) for all feasible X ≥ 0.
• X 0⇒ ‖X‖∗ = tr(X ) for all feasible X .
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
The planted case
• Consider graphs constructed as follows:
• Start with disjoint cliques C1, . . . ,Ck of size r1, . . . , rk .
• Noise: add set Ck+1 containing rk+1 additional nodes andadditional edges either deterministically by an adversary or atrandom independently with fixed prob p.
• C1, . . . ,Ck induce a feasible solution of (∗):
X ∗ =k∑
i=1
1
riviv
Ti
where vi ∈ RV is the characteristic vector of Ci .
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Results
• if not too much extra noise is added then, we can recover X ∗
(and hence C1, . . . ,Ck) by solving (∗).
• For both formulations, can add at most rk+1 = O(r2) extranodes, where r = mini=1,...,k ri .
• Adv case: Can add up to O(r2) additional edges, providedthere is at most O(minrq, rcl(v)) edges from v to Cq.
• Random case: “too many” noise edges quantified by numberof cliques k and the discrepancy between their sizes: need(
k∑s=1
rs2
)1/2 k∑
q=1
1
rq
1/2
≤ O(r).
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
The maximum mean weight KDC problem
Mean weight k-disjoint-clique problem (WKDC):
Given a complete graph KN and edge-weights W , find ak-disjoint-clique subgraph of KN that maximizes the sum ofaverage weights covered by each clique.
We relax WKDC as the SDP
max tr(WX )s.t. Xe ≤ e,
(w∗) X ≥ 0,tr(X ) = k,X 0
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Weighted planted case
• Unlike KDC, planted case does not correspond to anyparticular structure in input graph.
• Instead, is induced by edge-weight matrix W : entries of Wcorresponding to the planted k-disjoint-clique subgraph arelarger than the rest.
• Let C1, . . . ,Ck ⊆ V define a k-disjoint-clique subgraph ofthe complete graph KN = (V ,E ) on N vertices.
• Randomly sample entries of W from two distributions Ω1,Ω2
such that if u, v in same clique Ci then E [Wuv ] = α;otherwise E [Wuv ] = β for α > β.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Results: weighted case
• If not too much noise, the relaxation (w∗) of WKDC is exactwith extremely high probability for sufficiently large r .
• Can tolerate up to O(r) additional nodes.
• As before, too much edge noise is quantified by number ofcliques k and the discrepancy between their sizes:(
k+1∑s=1
rs
)1/2
≤ O(r).
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Rehnquist Supreme Court
• Data set is the set of U.S. Supreme Court Justices (servingfrom 1994-95 to 2003-04).
• Assign edge-weights corresponding to fraction of decisions onwhich Justices agreed:
St Br Gi So Oc Ke Re Sc Th1 St 1 0.62 0.66 0.63 0.33 0.36 0.25 0.14 0.152 Br 0.62 1 0.72 0.71 0.55 0.47 0.43 0.25 0.243 Gi 0.66 0.72 1 0.78 0.47 0.49 0.43 0.28 0.264 So 0.63 0.71 0.78 1 0.55 0.5 0.44 0.31 0.295 Oc 0.33 0.55 0.47 0.55 1 0.67 0.71 0.54 0.546 Ke 0.36 0.47 0.49 0.5 0.67 1 0.77 0.58 0.597 Re 0.25 0.43 0.43 0.44 0.71 0.77 1 0.66 0.688 Sc 0.14 0.25 0.28 0.31 0.54 0.58 0.66 1 0.799 Th 0.15 0.24 0.26 0.29 0.54 0.59 0.68 0.79 1
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Rehnquist Supreme Court: results
• We solved WKDC with k = 2 using SeDuMi. We obtainedthe following partition of the supreme court:
1: “Liberal” 2: “Conservative”
Stevens (St) O’Connor (Oc)Breyer (Br) Kennedy (Ke)Ginsberg (Gi) Rehnquist (Re)Souter (So) Scalia (Sc)
Thomas (Th)
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Rehnquist Supreme Court: results
• Algorithm is sensitive to choice of k .
• Solve with k = 3:
1: “Most Conservative” 2: “Moderate Conservative” 3: ”Liberal”
Thomas (Th) O’Connor (Oc) Stevens (St)Scalia (Sc) Kennedy (Ke) Breyer (Br)
Rehnquist (Re) Ginsberg (Gi)Souter (So)
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Image Segmentation
• Want to partition pixels of a given image into similar“segments” (wrt some property, eg. intensity).
• Essentially want to cluster the pixels of the image.
• Solve WKDC using with edge weights given by
Wij = exp(−‖Fi − Fj‖2/σF − ‖Xi − Xj‖2/σX )
where Fi is the feature value, Xi is the position in image ofthe ith pixel, parameters σF , σX chosen by user.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
• Example 238011 from the Berkeley Segmentation Data Set.• Original image 300x300; work with resized image 30x30.
Takes 4321s to solve.• Fi is RGB value of ith pixel, σF = 0.3, σX = 10000.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
• Example 135069 from the Berkeley Segmentation Data Set.• Original image 175x230; resized image 25x33. Takes 3430s to
solve.• Fi is RGB value of ith pixel, σF = 0.3, σX = 10000.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Conclusions
• The clique and clustering problems be relaxed as SDPs.
• We obtain the exact solution in polynomial time if the datacontains the desired hidden structure.
• Future work:• extension to other problems: eg. other graph-partitioning
objectives (e.g. normalized cut).• faster algorithms: more efficient solvers for SDP/doubly
nonnegative progams and/or theoretical guarantees for spectralclustering heuristics.
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis
Clusters, cliques, and low-rank matrices The maximum node KDC problem The weighted KDC problem Numerical results
Thank you!
• B. Ames and S. Vavasis (2010). Convex optimization for theplanted k-disjoint-clique problem arXiv:1008.2814
• B. Ames and S. Vavasis (2009). Nuclear norm minimizationfor the planted clique and biclique problems arXiv:0901.3348
Semidefinite Relaxation for the Clique Partitioning and Clustering Problems B. Ames and S. Vavasis