Approaching optimality [FOCS10]
An O(mlog2 n) algorithm for solving SDD systems
Yiannis Koutis , U of Puerto Rico, Rio PiedrasGary Miller, Richard Peng, CMU
Optimality achieved…probablyAn O(mlog n) algorithm for solving SDD systems
Yiannis Koutis , U of Puerto Rico, Rio PiedrasGary Miller, Richard Peng, CMU
Our motivation: Very large sparse linear systems
Ax = b: dimension n, m non-zero entries
Lower Bound O(m)Matrix inversion O(n!)Symmetric positive definite matrices: O(m n)Planar positive definite matrices: O(n1.5)
[LRT79]
Planar non-singular matrices: O(n1.5) [AY10]Many open problems!
Symmetric, negative off-diagonals, zero row-sums.
SDD matrix: Add a positive diagonal to Laplacian and flip off-diagonal signs, keeping symmetry.
1
Laplacian and SDD matrices
2
2015
30
1
Our motivation: Very large sparse SDD linear systems
Spielman and Teng 04: General SDD linear systems can be solved in time O(m log25 n). Planar systems can be solved in time O(m log2 n).
KM07: Planar SDD systems can be solved in time O(m).
KMP10--11: SDD linear systems in time O(m log n) .
(up to lower order terms and error, probability of failure)
A powerful algorithmic primitive
The Laplacian paradigm(or, why care for SDD systems? )
Classical scientific computing dealt with SDD’s since the 70s. Certain systems were known to be solvable in linear time.
Vaidya sent a paper to FOCS or STOC in 90-91
Vaidya then had a different idea
CASI is now the main solver provider to
(market cap 3.45B)
REJECT
The Laplacian paradigm(or, why care for SDD systems? )
The solver is the main routine in the fastest known algorithms for:
Approximate maximum flow, minimum cut problem◦ Christiano, Kelner, Madry, Spielman, Teng 2010
Computation of a few fundamental eigenvectors◦ Spectral methods for image segmentation (Shi and Malik 00, Miller and
Tolliver 08)
◦ Analysis of protein structures (Liu, Eyal, Bahar 08)
Solving elliptic finite element systems◦ Boman, Hendrickson, Vavasis 04, Avron et.al. 2009
Generating a random spanning tree in a graph◦ Madry and Kelner 09
Max flow, generalized lossy flow problems◦ Daitch and Spielman 08
Things to take home #1
A probably optimal and potentially practical linear system solver
Central component in several algorithms
Solving almost as easy as sortingThink about re-formulating your
problem to include an SDD system
Preconditioning
The solver is based on preconditioning
Two contradictory goals:Systems with B must be ‘simpler’ than ACondition number must be
small
Rate of convergence depends on
Recursive Preconditioning
Start with matrix A1
Compute preconditioner B1
Greedy-factorize B1
The iteration needs to solve a system in B1 which can be transformed via L to A2
Use recursion on A2 : Solve it with a preconditioned method
A1
B1
A2
B2
A3
Preconditioning Chain
Things to take home #2A good ‘two-level’ preconditioning
algorithm implies a fast solver
The preconditioner B must satisfy:
(up to small constants)
FORGET SDD MATRICES
FOCUS ON LAPLACIANS AND GRAPHS
(SDD systems can be transformed to Laplacian systems)
Combinatorial Preconditioning‘Traditional’ preconditioners were
taken to be sub-matrices of the system.
Pravin Vaidya made the paradigm shift:
The preconditioner of a graph must be a graph
(support graph theory enables deeper understanding of condition numbers)
The edge weight corresponds to the capacity of the wire. The effective resistance between two nodes u and w
is equal to the voltage difference between nodes u and w that is necessary to drive one unit of electric flow between them.
Rayleigh’s Monotonicity Law: Dropping edges increases the effective resistance between any u and w.
Graphs as electrical networks
2
2015
30
11
Graph Sparsification:(the heart of preconditioning)
The Sparsification Conjecture (Spielman and Teng)
For every graph A there is a sparse graph B, such that:
Graph Sparsification:(the heart of preconditioning)
Spielman and Teng’s key theorem:
For any Laplacian A, there is a Laplacian B with O(nloga n) edges such that k(A,B)<2.
Furthermore B can be computed in nearly-linear O(nlogb n) time.
Spielman and Srivastava proved a strong theorem:
For any Laplacian A, there is a Laplacian B with O(nlog n) edges such that k(A,B)<2. The
graph B can be computed by solving O(log n) systems.
Spectral sparsifiers with O(n logn) edges
The algorithm – Simple Sampling procedure
1. Let pe a probability distribution over the edges
2. pe is proportional to we Re, :: Re is the effective resistance of e
3. Let t = e we Re (*it can be shown that t= n-1*)
4. Draw q=O(t log t ) samples from p◦ Each time an edge e is picked add e to B, with
weight we/(qpe)Proof based n a thorem of Rudeslon and Vershynin +
Linear Algebra
Things to take home #3
Focus on Laplacians/graphsPrecondition graphs by graphsThe key is graph sparsification
A very elegant sampling algorithm works
The sampling probabilities depend on the effective resistances, which require a linear system solution
Sampling using upper bounds(Input: Graph A, Output: graph B with )
Let pe a probability distribution over the edges
pe is proportional to we Re, where Re is the effective resistance of e
pe is proportional to we R’e where R’e > Re
Let t = e we Re (*it can be shown that t= n-1*)
Let t = e we R’e (*t must be as small as possible *)
Draw q=O(t log t ) samples from p◦ Each time an edge e is picked add e to B, with
weight we/(qpe)An equality in SS08, becomes an inequality, using Re < R’e
· (A;B) < 2
Fix a spanning tree T Let RT(e) be the effective resistance of the
edge e if we “throw out” all edges not in TRayleigh’s Monotonicity Law :RT(e) > Re
For fixed tree the RT(e)’s computed in linear time
The product RT(e)*we is the stretch of e by T
We want to minimize the total stretch
Effective resistances hard to compute?Compute good and fast bounds.
Low-strecth tree: The best tree preconditioner
Vaidya proposed a maximum spanning tree (MST) which gave the first non-trivial bound on condition number
Boman and Hendrickson pinned down the right catalytic idea:
A low-stretch tree The stretch of e over T
The condition of T and A
An edge and its weight in A
A unique path in Tand its effective
resistace
Spectral sparsifier with n+m/k edges?
It looks that number of samples is O(m log2n) despite the availability a low-stretch tree
However we can ‘make’ a related graph with a tree of even lower total stretch. How ?
Scale up the edges of the tree by a factor of k>1
This decreases the total stretch and the number of off-tree samples by a factor of k, but incurs a condition number of k.
Scaling and sampling proportionally to stretch
(Input: Graph A, Output: graph B with )
Let A’ = A + k copies of low-stretch tree T pe is proportional to we R(kT)e where R(kT)e is the
effective resistance between the endpoints of e in kT Let t = e we R(kT)e (*t is total stretch = n+ mlog n/k
*) Draw O(t log t ) samples from p
◦ Each time an edge e is picked add e to B, with weight we/(qpe)
Algorithm over-samples edges in T and samples mlog2n/k off-tree edges
Theorem: For each graph A with n nodes and m edges, there is
an incremental-sparsifier B that O(k)-approximates A and has n+mlog2 n /k edges. B can be computed in time O(m logn).
Solver follows from k=log4 n
OverviewIncremental sparsifier is computed by:
◦Computing low-stretch tree◦Scaling-up low-stretch tree◦Sampling with probabilities proportional to stretch
Incremental sparsification is applied iteratively, interleaved with greedy contractions to produce a preconditioning chain
Preconditioning chain is used with a standard recursive iterative method to solve system
O(m log n) solver?
O(m log2 n) solver computes a low-stretch tree for every graph in the preconditioning chain.
O(m logn) solver computes a low-stretch tree only for the top (input) graph. The same tree can be kept for the whole chain.
There are some complications. In usual chain the number of edges goes down by at least O(log2n) between every two graphs in the chain. Stagnation can occur in the new construction. But it all works out.
Open QuestionsParallelization ?
◦O(m logc n) work in O(n1/3) time [spaa11]
◦ Ideally O(m log n) work in O(log n) time
Practical implementations◦A practical low-stretch tree◦A practical sparsifier
Is it possible to compute sparsifier with O(n log n) edges more efficiently than solving systems?