On recent improvements in the conic optimizer in MOSEK

http://www.mosek.com

On recent improvements in theconic optimizer in MOSEK

Erling D. Andersen

MOSEK ApS,Fruebjergvej 3, Box 16,

2100 Copenhagen,Denmark.

Email: [email protected]: http://www.mosek.com

August 25, 2009

Introduction

2 / 22

What is conic optimization!

IntroductionWhat is conicoptimization!

Basic cone types

CommentsMOSEK and conicoptimization

Topics

An example

The algorithm

3 / 22

Primal problem:

(COP ) min cT xst Ax = b,

x ∈ K1 ×K2 × · · · × Kk,

where each Kk is a convex cone (closed, pointed and solid).Dual problem:

(COD) max bT yst AT y + s = c,

s ∈ K∗

1 ×K∗

2 × · · · × K∗

k.

where K∗

p is the dual cone corresponding to Kp.

Basic cone types


Basic cone types


Topics

An example

The algorithm

4 / 22

■ Linear:Kl := {x ∈ R : x ≥ 0}

■ Quadratic:

Kq :=

x ∈ Rn : x1 ≥

√

√

√

√

n∑

j=2

x2j

■ Rotated quadratic:

Kr :=

x ∈ Rn : 2x1x2 ≥n

∑

j=2

x2j , x1, x2 ≥ 0

Comments


Basic cone types


Topics

An example

The algorithm

5 / 22

■ All cones are self-dual and homogeneous (=symmetric).■ Many more interesting convex cones exists.

◆ Positive semi-definite.

■ The rotated cone is an (important) special case of thequadratic cone.

◆ Special structure that can be exploited.

■ Applications:

◆ Portfolio optimization (with transactions cost).◆ Convex quadratic optimization.◆ Robust optimization.◆ Image denoising.

MOSEK and conic optimization


Basic cone types


Topics

An example

The algorithm

6 / 22

■ MOSEK solves large-scale sparse problems.■ Version 1 released in 1999 includes CQO.■ Expirience with CQO:

◆ Widely used after a slow start.◆ Mostly finance related models.◆ No robust optimization.◆ Easy to teach the users.

■ Convexity is not an issue.■ Usually leads to good formulations.■ Easy to specify.

◆ More robust than traditional function based convexoptimization.

Topics


Basic cone types


Topics

An example

The algorithm

7 / 22

■ Some experiences.

◆ Is bad duality states an issue.

■ Developments since 1999 [Andersen, Roos, Terlakypaper].

■ Expected future improvements.

An example

8 / 22

Non-attainment in the dual problem

Introduction

An example

Non-attainment inthe dual problem

Example cont.

The algorithm

9 / 22

minimize x4

subject to x3 + x4 = 1,x1 = 0,x2 = 1,

2x1x2 ≥ x23,

x1, x2 ≥ 0,

(1)

Optimal solution:

x∗

1 = 0, x∗

2 = 1, x∗

3 = 0 and x∗

4 = 1

The dual:maximize y1 + y3

subject to y2 + s1 = 0,y3 + s2 = 0,y1 + s3 = 0,

y1 = 1,2s1s2 ≥ s2

3,s1, s2 ≥ 0.

(2)

Example cont.

Introduction

An example

Non-attainment inthe dual problem

Example cont.

The algorithm

10 / 22

Therefore,y∗1 = 1

ands∗3 = −1.

This implies that2s∗1s

∗

2 ≥ (s∗3)2 = 1

and hence s∗2 > 0. Given this fact we can conclude that

y∗1 + y∗3 = 1 − s∗2< 1

so in practice terminated at s∗2 = O(ε) and s∗1 = O(1/ε). Badduality states happens in practice! What to do?

The algorithm

11 / 22

The homogeneous model

Introduction

An example

The algorithm

The homogeneousmodel

One iterationNew stoppingcriteriaNew step-sizecomputation

An example

Other improvements

The futureComputationalresults

Feasible problems

Primal infeasibleproblems

Summary andconclusions

12 / 22

(H) Ax − bτ = 0,AT y + s − cτ = 0,

−cT x + bT y − κ = 0,(x; τ) ∈ K, (s;κ) ∈ K

■ All solutions satisfies: xT s + τκ = 0.■ Find a solution such that τ + κ > 0.■ Algorithmic framework:

◆ The Nesterov-Todd search direction.◆ Mehrotra predictor-corrector.◆ Plus other advanced stuff.

One iteration

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



13 / 22

NT search direction:

Adx − bdτ = η(Ax(0) − bτ (0)),AT dy + ds − cdτ = η(AT y(0) + s(0) − cτ (0)),

−cT dx + bT dy − dκ = η(−cT x(0) + bT y(0) − κ),X̄(0)T (ΘW )−1ds + S̄(0)TΘWdx = −X̄(0)S̄(0)e + γµ(0)e,

τ (0)dκ + κ(0)dτ = −τ (0)κ(0) + γµ(0).

where η := γ − 1 and γ ∈ [0, 1).New iterate:

x(1)

τ (1)

y(1)

s(1)

κ(1)

=

x(0)

τ (0)

y(0)

s(0)

κ(0)

+ α

dx

dτ

dy

ds

dκ

New stopping criteria

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



14 / 22

Optimal case:

∥

∥

∥Axk

τk − b∥

∥

∥ ≤ εp(1 + ‖b‖),∥

∥

∥AT xk

τk + sk

τk − c∥

∥

∥ ≤ εd(1 + ‖c‖),

min(

(xk)T sk+τkκk

(τk)2,∣

∣

∣

cT xk

τk − bT yk

τk

∣

∣

∣

)

≤ εg max(

1,∣

∣

∣

cT xk

τk

∣

∣

∣

)

,

Primal infeasible case:

εibT yk >

‖b‖

max(1, ‖c‖)

∥

∥

∥AT yk + sk∥

∥

∥

Dual infeasible case:

−εicT xk >

‖c‖

max(1, ‖b‖)

∥

∥

∥Axk∥

∥

∥

All εs are user specified.

New step-size computation

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



15 / 22

(x(0) + αdx, τ (0) + αdτ , s(0) + αds, κ

(0) + αdκ) ∈ N (β)

where β ∈ (0, 1) and N(β) is a suitable neighborhooddefinition.Comments:

■ Conic case pre version 6.

◆ Simple back-tracking line search.◆ Usually not close to maximum step-size.

■ Linear case

◆ 0.9995 times the distance to the boundary.◆ Generally accepted.

■ Conic version 6.

◆ 0.95 times the distance to the boundary.◆ Aggressive choice of step-size is not good.◆ But poor asymptotic convergence.

An example

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



16 / 22

0.95 fraction:

ITE KAP/TAU POBJ DOBJ MU TIME0 1.0e+000 1.000000000e+000 0.000000000e+000 1.0e+000 0.031 5.0e-002 5.000000000e-002 0.000000000e+000 5.0e-002 0.072 2.5e-003 2.500000000e-003 0.000000000e+000 2.5e-003 0.073 1.2e-004 1.250000000e-004 0.000000000e+000 1.3e-004 0.084 6.2e-006 6.250000000e-006 0.000000000e+000 6.3e-006 0.085 3.1e-007 3.125000000e-007 0.000000000e+000 3.1e-007 0.096 1.6e-008 1.562500000e-008 0.000000000e+000 1.6e-008 0.097 7.8e-010 7.812500000e-010 0.000000000e+000 7.8e-010 0.09

0.99995 faction:

ITE KAP/TAU POBJ DOBJ MU TIME0 1.0e+000 1.000000000e+000 0.000000000e+000 1.0e+000 0.281 5.0e-005 5.000000000e-005 0.000000000e+000 5.0e-005 0.372 2.5e-009 2.500000000e-009 0.000000000e+000 2.5e-009 0.37

Comments:

■ Room for improvements.

Other improvements

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



17 / 22

■ Cleaned up the implementation.■ Improved factor, dense column detection etc.■ Improved presolve.

The future

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



18 / 22

■ Improve presolve on conic part.■ An automatic dualizer.■ Exploiting intersection of cones structure. (Talked about

that at ISMP 2003).■ Improve performance on mixed integer problems.

Computational results

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



19 / 22

■ Computer:

Dell PowerEdge R410Intel Xeon L5520 (2.26GHz, 8M Cache)16GB Memory for 2CPU (4x4GB Dual Rank RDIMMs) 1066Mhz

■ OS: 64 bit CentOS Linux.■ MOSEK version 6.0.0.32.■ MOSEK version 5.0.0.125.

Feasible problems

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



20 / 22

■ Small problem: Fastest optimizer ≤ 6s■ Small problem: 6s < Fastest optimizer ≤ 60s■ Ratio

time v6+0.01

time v5 + 0.01.

small medium large6 5 6 5 6 5

Num. 190 190 38 38 9 9Firsts 153 93 28 10 9 0Total time 138.38 414.08 737.19 1325.03 793.56 1141.30G. avg. 0.21 0.23 15.16 21.58 82.45 105.37G. avg. r 0.89 0.70 0.78

■ 10% improvement on average for small problems.■ 20% improvement on average for medium to large sized

problems.■ Much improved stability.

Primal infeasible problems

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



21 / 22

Obj. Status Iter. Time1 9.0057488155e-01 p. infs. 8 0.02 -2.1286428024e-03 p. infs. 15 0.03 -6.8372548723e+02 p. infs. 19 0.04 6.1809772127e-03 p. infs. 0 0.05 4.0000000000e+00 p. infs. 0 0.06 -2.2498965504e+03 p. infs. 16 0.07 -1.9942649941e+03 p. infs. 17 0.08 -3.4833362961e+03 p. infs. 16 0.09 -2.9317422673e+03 p. infs. 17 0.010 9.5422268727e-04 p. infs. 25 0.0

■ Smallish problems.■ Obj = bT y. (Negative values for max problems.)■ Detects infeasibility reliably.

Summary and conclusions

Introduction

An example

The algorithm



An example

Other improvements


Feasible problems



22 / 22

■ Bad duality states happens in practice.■ Version 6 is a significant improvement of version 5.■ The homogenous model and alg. detects infeasibility.■ Much work still to be done.

Documents

On recent improvements in the conic optimizer in MOSEK