Exploiting Structure in SDPs with Chordal Sparsityaaa/Public/Presentations/CDC_2016_Papachristodoulou.pdf · Exploiting Structure in SDPs with Chordal Sparsity AntonisPapachristodoulou

Exploiting Structure in SDPs with Chordal Sparsity

Antonis PapachristodoulouDepartment of Engineering Science, University of Oxford

Joint work with Yang Zheng, Giovanni Fantuzzi, Paul Goulart and Andrew Wynn

CDC 2016 Pre-conference Workshop

OUTLINE

22 ADMM for Primal and Dual Sparse SDPs

11 Chordal Graphs and Positive Semidefinite Matrices

33 CDCS: Cone Decomposition Conic Solver

44 Conclusion

graph ( , ) is chordal if every cycle of 4 chorlen hagth .ds a A =

≥G V E

1

2 3

4 12

3

45

6

7

8

910

11

12

Can recognise chordal graphs in (| | | |) timeO +V E

Chordal Graphs

1

2 3

4 12

3

45

6

7

8

910

11

12

Chordal Graphs

Can recognise chordal graphs in (| | | |) timeO +V E

graph ( , ) is chordal if every cycle of 4 chorlen hagth .ds a A =

≥G V E

1

A maximal clique is a clique that is nota subset of another clique

{1,2,6}.

.e.g. =C

1

2 3

4

56

C1C3

C2

C4

Can find the maximal cliques of a chordal graph in (| | | |) time.O +V E

Maximal Cliques

1

2 3

4

56

1C

2C

3C

4C

a)

b)

c)

d)

Examples of Chordal Graphs

{ }( ,0) | 0, ( , )n nijZ Z i jZ = ∈ = ∀ ∉∈ SS E E

( ,0) { 0}( ,0) |n n ZZ+ ∈=E ES S

41

2 3

4

56

( ) ( , )Z =G V E

Z

z11

z12

0 0 0 z16

z12

z22

z23

z24

0 z26

0 z23

z33

z34

0 00 z

24z

34z

44z

45z

46

0 0 0 z45

z55

z56

z16

z26

0 z46

z56

z66

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥⎥

Sparse Matrices

1 2

1 2

{ , , , }. Suppose

tha

Theorem: Let ( , ) be a chordal graph withset of maximal cliques

( ,0). Then ( ,0) if and only ifthere exists a set of matrices

t { , , , } such

t

hat ,

n

k

n

p

p

k

ZZ

Z Z Z

Z Z Z

+

= …

∈ ∈

=

=

…

S SC C C C

G V E

E E

10, 0 if ( , .)

pkij k k

kZ i j

=

= ∈ ×/∑ E E

Agler’s Theorem

1

2 3

4

56

1C

Example: Applying Agler’s Theorem

2C

3C

4C

4

5

3

1

2

6

2

6

4

2

6

4

Example: Applying Agler’s Theorem

2C

3C

4C

1C

• Applications: control theory, fluid mechanics, machine learning, polynomial optimization, combinatorics, operations research, finance, etc.

• Second-order solvers : SeDuMi, SDPA, SDPT3 • Large-scale cases: exploit the inherent structure of the

instances (De Klerk, 2010):Low Rank and Algebraic Symmetry.Chordal Sparsity:

Second-order methods: Fukuda et al., 2001; Nakata et al., 2003; Andersen et al., 2010;First-order methods: Madani et al., 2015; Sun et al., 2014.

1. SDPs with Chordal Sparsity

Dual

SDPs with Chordal Sparsity

min ,

subject to ( )

X

n

C X

X bX +

=∈S

A,

max ,

subject to ( ) Z

y Z

n

b y

y CZ

∗

+

+ =∈SA

Primal SDP Dual SDP

DecomposedDual SDPDuality

Grone’sTheorem

ChordalGraphs

DecomposedPrimal SDP

Duality

Agler’sTheorem

Decomposing Sparse SDPs

1. SDPs with Chordal SparsitySDPs with Chordal Sparsity

( , )=G V E

{ }( ,0) | 0, ( , )n nijX X i j= ∈ = ∀ ∉S SE E

( ,0) { 0}( ,0) |n n XX+ ∈=E ES S

( ,?) partial symmetric matrices with entries defined on n n n= ×S E E

( ,?) ( ,?) | 0, , ( , ){ }n nij ijM M XX i j+ = ∃ ≥ = ∀∈ ∈S SE E E

( ,?) ( ,0) are dual cones o an f each ot ed h rn n+ +E ES S

1. SDPs with Chordal Sparsity

Grone’s Theorem:

i.e., matrix X is PSD

completable

Grone’s theorem

1

Consider a choral graph , with a set of maximal cliques ,

(, .

)

p

=G V EC C…

( ,?) if and only if ( ) 0, 1, , .nkX X k p+∈ = …S E C

1( ) 0X C

2( ) 0X C

3( ) 0X C

( ,?)nX +∈S E

,

1

max ,

s.t ( )

Z , 1, ,

k k

k

y Z

pT

kk

b y

y E Z E C

k p

∗

=

+

+ =

∈ = …

∑S

C C

C

A

min ,

s.t. ( ) , 1, ,k

k k

X

T

C X

X bE XE k p+

=∈ = …S C

C C

A

,max ,

subject to ( ) Z

y Z

n

b y

y CZ

∗

+

+ =∈SA

min ,

subject to ( )

X

n

C X

X bX +

=∈S

A

2. ADMM for Primal and Dual Sparse SDPs

Primal Dual

Cone Decomposition of Primal and Dual SDPs

( ,?)nX +∈S E ( ,0)nZ +∈S E

OUTLINE




44 Conclusion

2. ADMM for Primal and Dual Sparse SDPsAlternating Direction Method of Multipliers

min ( ) ( )s.t.

f x g zx z

+=

2

1

2

2

1 1

2

1 1 1

1argmin ( )2

1argmin ( )2

( )

n n n

x

n n n

z

n n n n

x f x x z

z g z x z

x z

ρ λρ

ρ λρ

λ λ ρ

+

+ +

+ + +

⎛ ⎞= + − +⎜ ⎟⎜ ⎟

⎝ ⎠⎛ ⎞

= + − +⎜ ⎟⎜ ⎟⎝ ⎠

= + −

An operator splitting method for a problem of the form

f, g may be nonsmooth.

ADMM:

2

2

1( ) ( )2

f x g z x zρ λρ

= + + − +LLagrangian:

2. ADMM for Primal and Dual Sparse SDPsReformulation and decomposition of the PSD constraint

min ,s.t. ( ) , 1, , , 1, ,

k k

k

X

Tk

k

C XX b

X E XE k pX k p+

== = …∈ = …

C C

C

A

S

1, , ,min s.t. , 1, , , 1, ,

p

T

x x x

k k

k k

c xAx bx H x k px k p

== = …∈ = …S

…

• Using indicator functions

• Augmented Lagrangian

• Regroup the variables

10, ,

1

min ( ) ( )

s.t. , 1, ,

kp

pT

kx x xk

k k

c x Ax b x

x H x k p

δ δ=

+ − +

= = …

∑ S…

2

01

1( ) ( )2k

pT

k k k kk

c x Ax b x x H xρδ δ λρ=

⎡ ⎤= + − + + − +⎢ ⎥

⎢ ⎥⎣ ⎦∑ SL

{ } { } { }1 1; , ; ,p px x x λ λ… …X Y Z


1) Minimization over

2) Minimization over

3) Update multipliers

Projections in parallel

QP with linear constraint

the KKT system matrix only depends on the problem data.

ADMM for Primal SDPs

2

( ) ( )

1

1min 2

s.t.

pT n n

k k kxk

c x x H x

Ax b

ρ λρ=

+ − +

=

∑

2

( 1) ( )1min

s.t.

k

n nk k kx

k k

x H x

x

λρ

+− +

∈S

( )( 1) ( ) ( 1) ( 1)n n n nk k k kx H xλ λ ρ+ + += + −

2

01

1( ) ( )2k

pT

k k k kk

c x Ax b x x H xρδ δ λρ=

⎡ ⎤= + − + + − +⎢ ⎥

⎢ ⎥⎣ ⎦∑ SL

{ } { } { }1 1; , ; ,p px x x λ λ… …X Y Z

X

Y


The duality between the primal and dual SDP is inherited by the decomposed problems by virtue of the duality between Grone’s and Agler’s theorems.

ADMM for Primal and Dual SDPs

,z

1

min

s.t.

z 0, 1, , , 1, ,

k

T

yp

T Tk k

k

k k

k k

b y

A y H v c

v k pz k p

=

−

+ =

− = = …∈ = …

∑

S

,min s.t. , 1, ,

k

T

x x

k k

k k

c xAx bx H xx k p

==∈ = …S

3. ADMM for the Homogenous Self-dual Embedding

• Notational simplicity

• KKT conditionsPrimal feasible

Dual feasible

Zero-duality gap

ADMM for the Homogeneous self-dual embedding

1 1 1 1

, , ,

p p p p

x z v Hs z v H

x z v H

⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦

, 0,, 0,

Ax r b rs w Hx w s

∗ ∗ ∗

∗ ∗ ∗ ∗ ∗

− = =+ = = ∈S

, 0,0,

T TA y H v h c hz v z

∗ ∗ ∗ ∗

∗ ∗ ∗

+ + = =− = ∈S

0T Tc x b y∗ ∗− =

3. ADMM for the Homogenous Self-dual Embedding

• Notational simplicity

• Feasibility problem


0 00 0 0 0

0 0 00 0 0

0 0 0

T T

T T

h xA H cz sIr yA bw vH I

c bκ τ

− −⎡ ⎤⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥

= ⎢ ⎥⎢ ⎥ ⎢ ⎥−⎢ ⎥⎢ ⎥ ⎢ ⎥−⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥−⎣ ⎦ ⎣ ⎦⎣ ⎦

, are two non-negativeand complementary variables

τ κ

2

0 00 0 0 0

, , , 0 0 00 0 0

0 0 0

d

T T

nn m

T T

h x A H cz s I

v u Qr y A bw v H I

c bκ τ

+

− −⎡ ⎤⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥

= × × × ×⎢ ⎥⎢ ⎥ ⎢ ⎥ −⎢ ⎥⎢ ⎥ ⎢ ⎥ −⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥ −⎣ ⎦ ⎣ ⎦ ⎣ ⎦

K S

find ( , )s.t. ( , )

u vv Quu v ∗

=∈ ×K K

Block elimination can be applied here to speed up the projection;Then, the per-iteration cost is the same as applying a splitting method to the primal or dual alone.

[1] O’Donoghue, B., Chu, E., Parikh, N. and Boyd, S. (2016). Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications, 169(3), 1042– 1068

• ADMM steps (similar to solver SCS [1])

Q is highly structured and sparse

Projection to a subspaceProjection to cones


find ( , )s.t. , ( , )

u vv Qu u v ∗= ∈ ×K K

1 1

1 1

1 1 1

ˆ ( ) ( )ˆP ( )ˆ

k k k

k k k

k k k k

u I Q u vu u vv v u u

+ −

+ +

+ + +

= + += −= − +

K

0 00 0 0 0

0 0 00 0 0

0 0 0

T T

T T

A H cI

Q A bH Ic b

− −⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥−⎢ ⎥−⎢ ⎥⎢ ⎥−⎣ ⎦

OUTLINE




44 Conclusion

4. CDCS: Cone Decomposition Conic Solver

• An open source MATLAB solver for partially decomposable conic programs;

• CDCS supports constraints on the following cones:Free variablesnon-negative orthantsecond-order conethe positive semidefinite cone.

• Input-output format is aligned with SeDuMi;• Works with latest YALMIP release.

Download from https://github.com/OxfordControl/CDCS

[x,y,z,info] = cdcs(At,b,c,K,opts);

Syntax:

CDCS: Cone Decomposition Conic Solver

4. CDCS: Cone Decomposition Conic SolverRandom SDPs with block-arrow pattern

• Block size: d• Number of Blocks: l• Arrow head: h• Number of constraints: m

Numerical Comparison• SeDuMi• sparseCoLO+SeDuMi• SCS• sparseCoLO+SCS= 10CDCS and SCSNumerical Result


32

4. CDCS: Cone Decomposition Conic SolverBenchmark problems in SDPLIB

Three sets of benchmark problems in SDPLIB (Borchers, 1999):

1) Four small and medium-sized SDPs ( theta1, theta2, qap5 and qap9);

2) Four large-scale sparse SDPs (maxG11, maxG32, qpG11 and qpG51);

3) Two infeasible SDPs (infp1 and infd1).


4. CDCS: Cone Decomposition Conic SolverResult: small and medium-sized instances


4. CDCS: Cone Decomposition Conic SolverResult: large-sparse instances


4. CDCS: Cone Decomposition Conic SolverResult: Infeasible instances


4. CDCS: Cone Decomposition Conic SolverResult: CPU time per iteration

Our codes are currently written in MATLAB SCS is implemented in C.


5. Conclusion

• Introduced a conversion framework for sparse SDPs• Developed efficient ADMM algorithms

Primal and dual standard form;The homogeneous self-dual embedding;

Ongoing work• Develop ADMM algorithms for sparse SDPs arising in SOS.• Applications in networked systems and power systems

suitable for first-order methods

Conclusion

Thank you for your attention!

1. Zheng, Y., Fantuzzi G., Papachristodoulou A., Goulart, P., and Wynn, A. (2016) Fast ADMM for Semidefinite Programs with Chordal Sparsity. arXiv preprint arXiv:1609.06068

2. Zheng, Y., Fantuzzi, G., Papachristodoulou, A., Goulart, P., & Wynn, A. (2016) Fast ADMM for homogeneous self-dual embeddings of sparse SDPs. arXiv preprint arXiv:1611.01828

• CDCS: Download from https://github.com/OxfordControl/CDCS

Documents

Exploiting Structure in SDPs with Chordal Sparsityaaa/Public/Presentations/CDC_2016_Papachristodoulou.pdf · Exploiting Structure in SDPs with Chordal Sparsity AntonisPapachristodoulou