cvx_dcp

7/24/2019 cvx_dcp

1/44

Constructive Convex Analysis

and Disciplined Convex Programming

Stephen Boydand Steven Diamond

EE & CS Departments

Stanford University

MLSS, Kyoto, August 23-24 2015

1

7/24/2019 cvx_dcp

2/44

Outline

Convex Optimization


Disciplined Convex Programming

Modeling Frameworks

Conclusions

2

7/24/2019 cvx_dcp

3/44

Outline

Convex Optimization



Modeling Frameworks

Conclusions

Convex Optimization 3

7/24/2019 cvx_dcp

4/44

Convex optimization problem standard form

minimize f0(x)subject to fi(x) 0, i=1, . . . , m

Ax=b

with variable x Rn

objective and inequality constraints f0, . . . , fm are convex

for all x, y, [0, 1],

fi(x+ (1

)y)

fi(x) + (1

)fi(y)

i.e., graphs officurve upward

equality constraints are linear


7/24/2019 cvx_dcp

5/44

Convex optimization problem conic form

cone program:

minimize cTxsubject to Ax=b, x K

with variable x

Rn

linear objective, equality constraints;K is convex cone special cases:

linear program (LP) semidefinite program (SDP)

the modern canonical form

there are well developed solvers for cone programs


7/24/2019 cvx_dcp

6/44

Why convex optimization?

beautiful, fairly complete, and useful theory solution algorithms that work well in theory and practice

convex optimization is actionable

many applications in

control combinatorial optimization signal and image processing communications, networks circuit design machine learning, statistics finance

...and many more


7/24/2019 cvx_dcp

7/44

How do you solve a convex problem?

use an existing custom solver for your specific problem

develop a new solver for your problem using a currentlyfashionable method

requires work but (with luck) will scale to large problems

transform your problem into a cone program, and use a

standard cone program solver can be automatedusing domain specific languages


7/24/2019 cvx_dcp

8/44

Outline

Convex Optimization



Modeling Frameworks

Conclusions

Constructive Convex Analysis 8

7/24/2019 cvx_dcp

9/44

Curvature: Convex, concave, and affine functions

f is concave iff is convex, i.e., for any x, y, [0, 1],

f(x+ (1 )y) f(x) + (1 )f(y) f is affineif it is convex and concave, i.e.,

f(x+ (1

)y) =f(x) + (1

)f(y)

for any x, y, [0, 1] f is affine it has form f(x) =aTx+b


7/24/2019 cvx_dcp

10/44

Verifying a function is convex or concave

(verifying affine is easy)

approaches:

via basic definition (inequality) via first or second order conditions, e.g.,2f(x) 0

via convex calculus: construct f using library of basic functions that are convex or concave calculus rules or transformations that preserve convexity


7/24/2019 cvx_dcp

11/44

Convex functions: Basic examples

xp (p 1 orp 0), e.g., x2, 1/x (x>0) ex

xlog x

aTx+b

xTPx (P 0)x (any norm) max(x1, . . . , xn)


7/24/2019 cvx_dcp

12/44

Concave functions: Basic examples

xp (0 p 1), e.g.,x log x

xy xTPx (P 0) min(x1, . . . , xn)


7/24/2019 cvx_dcp

13/44

Convex functions: Less basic examples

x2/y (y>0), xTx/y (y>0), xTY1x (Y 0) log(ex1 +

+exn)

f(x) =x[1]+ +x[k] (sum of largest k entries) f(x, y) =xlog(x/y) (x, y>0)

max(X) (X=XT)


7/24/2019 cvx_dcp

14/44

Concave functions: Less basic examples

log det X, (det X)1/n (X 0) log (x) ( is Gaussian CDF) min(X) (X=X

T)


7/24/2019 cvx_dcp

15/44

Calculus rules

nonnegative scaling: f convex, 0 = f convex sum: f, g convex = f +g convex affine composition: f convex = f(Ax+b) convex pointwise maximum: f1, . . . , fm convex = maxifi(x) convex composition: h convex increasing, f convex = h(f(x)) convex

. . . and similar rules for concave functions

(there are other more advanced rules)


7/24/2019 cvx_dcp

16/44

Examples

from basic functions and calculus rules, we can show convexity of . . .

piecewise-linear function: maxi=1....,k(aTi x+bi)

1-regularized least-squares cost:

Ax

b2

2

+

x1, with

0

sum of largest kelements ofx: x[1]+ +x[k] log-barrier:mi=1log(fi(x)) (on{x| fi(x)< 0}, fi convex) KL divergence: D(u, v) = i(uilog(ui/vi) ui+vi) (u, v>0)


7/24/2019 cvx_dcp

17/44

A general composition rule

h(f1(x), . . . , fk(x)) is convex when h is convex and for each i

h is increasing in argument i, and fi is convex, or

h is decreasing in argument i, and fi is concave, or

fi is affine

theres a similar rule for concave compositions(just swap convex and concave above)

this one rule subsumes all of the others this is pretty much the only rule you need to know


7/24/2019 cvx_dcp

18/44

Example

lets show that

f(u, v) = (u+1) log((u+1)/ min(u, v))

is convex

u, vare variables with u, v>0

u+1 is affine; min(u, v) is concave

since xlog(x/y) is convex in (x, y), decreasing in y,

f(u, v) = (u+1) log((u+1)/ min(u, v))

is convex


7/24/2019 cvx_dcp

19/44

Example

log(eu1 + +euk) is convex, increasing so if f(x, ) is convex in x for each and >0,

log ef(x,1) + +ef(x,k) /k

is convex

this is log E ef(x,), where U({1, . . . , k}) arises in stochastic optimization via bound

log Prob(f(x, ) 0) log E ef(x,)


7/24/2019 cvx_dcp

20/44

Constructive convexity verification

start with function given as expression build parse tree for expression

leaves are variables or constants/parameters

nodes are functions of children, following general rule tag each subexpression as convex, concave, affine, constant

variation: tag subexpression signs, use for monotonicitye.g., ()2 is increasing if its argument is nonnegative

sufficient (but not necessary) for convexity


7/24/2019 cvx_dcp

21/44

Example

forx

7/24/2019 cvx_dcp

22/44

Example

analyzed by dcp.stanford.edu (Diamond 2014)


7/24/2019 cvx_dcp

23/44

Example

f(x) = 1+x2 is convex

but cannot show this using constructive convex analysis (leaves) 1 is constant, x is affine

x

2

is convex 1+x2 is convex but

1+x2 doesnt match general rule

writing f(x) = (1, x)2, however, works (1, x) is affine (1, x)2 is convex


7/24/2019 cvx_dcp

24/44

Outline

Convex Optimization



Modeling Frameworks

Conclusions

Disciplined Convex Programming 24

7/24/2019 cvx_dcp

25/44

Disciplined convex programming (DCP)

(Grant, Boyd, Ye, 2006)

framework for describing convex optimization problems

based on constructive convex analysis sufficient but not necessary for convexity

basis for several domain specific languages and tools forconvex optimization


7/24/2019 cvx_dcp

26/44

Disciplined convex program: Structure

a DCP has

zero or one objective, with form minimize{scalar convex expression} or maximize{scalar concave expression}

zero or more constraints, with form {convex expression} = {convex expression} or

{affine expression} == {affine expression}


7/24/2019 cvx_dcp

27/44

Disciplined convex program: Expressions

expressions formed from variables, constants/parameters,

andfunctions from a library library functions have known convexity, monotonicity, and

sign properties

all subexpressions match general composition rule


7/24/2019 cvx_dcp

28/44

Disciplined convex program

a valid DCP is convex-by-construction (cf. posterior convexity analysis) syntactically convex (can be checked locally)

convexity depends only on attributesof library functions,and not their meanings

e.g., could swap and 4, or exp and ()+, since their

attributes match


7/24/2019 cvx_dcp

29/44

Canonicalization

easy to build a DCP parser/analyzer

not much harder to implement a canonicalizer, whichtransforms DCP to equivalent cone program

then solve the cone program using a generic solver

yields a modeling frameworkfor convex optimization


7/24/2019 cvx_dcp

30/44

Outline

Convex Optimization



Modeling Frameworks

Conclusions

Modeling Frameworks 30

7/24/2019 cvx_dcp

31/44

Optimization modeling languages

domain specific language (DSL) for optimization express optimization problem in high level language

declare variables form constraints and objective solve

long history: AMPL, GAMS, . . . no special support for convex problems very limited syntax callable from, but not embedded in other languages


7/24/2019 cvx_dcp

32/44

Modeling languages for convex optimization

all based on DCP

YALMIP Matlab Lofberg 2004

CVX Matlab Grant, Boyd 2005

CVXPY Python Diamond, Boyd 2013Convex.jl Julia Udell et al. 2014

some precursors

SDPSOL (Wu, Boyd, 2000) LMITOOL (El Ghaoui et al., 1995)


7/24/2019 cvx_dcp

33/44

CVX

cvx_beginvariable x(n) % declare vector variable

minimize sum(square(A*x-b)) + gamma*norm(x,1)

subject to norm(x,inf)

7/24/2019 cvx_dcp

34/44

Some functions in the CVX library

function meaning attributesnorm(x, p) xp, p 1 cvxsquare(x) x2 cvx

pos(x) x+ cvx, nondecr

sum_largest(x,k) x[1]+ +x[k] cvx, nondecrsqrt(x)

x, x 0 ccv, nondecr

inv_pos(x) 1/x, x>0 cvx, nonincr

max(x) max{x1, . . . , xn} cvx, nondecrquad_over_lin(x,y) x2/y, y>0 cvx, nonincr in y

lambda_max(X) max(X), X=XT cvx


7/24/2019 cvx_dcp

35/44

DCP analysis in CVX

cvx_begin

variables x y

square(x+1)

7/24/2019 cvx_dcp

36/44

CVXPY

from cvxpy import *

x = Variable(n)

cost = sum_squares(A*x-b) + gamma*norm(x,1)

prob = Problem(Minimize(cost),

[norm(x,"inf")

7/24/2019 cvx_dcp

37/44

DCP l i i CVXPY

7/24/2019 cvx_dcp

38/44

DCP analysis in CVXPY

expr = (x y)2

1 max(x, y)

x = Variable()

y = Variable()

expr = quad_over_lin(x - y, 1 - max_elemwise(x,y))

expr.curvature # CONVEX

expr.sign # POSITIVEexpr.is_dcp() # True


P t i CVXPY

7/24/2019 cvx_dcp

39/44

Parameters in CVXPY

symbolic representations of constants can specify sign

change value of constant without re-parsing problem

for-loop style trade-off curve:

x_values = []

for val in numpy.logspace(-4, 2, 100):

gamma.value = val

prob.solve()x_values.append(x.value)


P ll l t l t d ff

7/24/2019 cvx_dcp

40/44

Parallel style trade-off curve

# Use tools for parallelism in standard library.from multiprocessing import Pool

# Function maps gamma value to optimal x.

def get_x(gamma_value):

gamma.value = gamma_valueresult = prob.solve()

return x.value

# Parallel computation with N processes.

pool = Pool(processes = N)

x_values = pool.map(get_x, numpy.logspace(-4, 2, 100))


Convex jl

7/24/2019 cvx_dcp

41/44

Convex.jl

using Convexx = Variable(n);

cost = sum_squares(A*x-b) + gamma*norm(x,1);

prob = minimize(cost, [norm(x,Inf)

7/24/2019 cvx_dcp

42/44

Outline

Convex Optimization



Modeling Frameworks

Conclusions

Conclusions 42

Conclusions

7/24/2019 cvx_dcp

43/44

Conclusions

DCP is a formalization of constructive convex analysis simple method to certify problem as convex

(sufficient, but not necessary) basis of several DSLs/modeling frameworks for convex

optimization

modeling frameworks make rapid prototyping of convexoptimization based methods easy

Conclusions 43

References

7/24/2019 cvx_dcp

44/44

References

Disciplined Convex Programming(Grant, Boyd, Ye)

Graph Implementations for Nonsmooth Convex Programs

(Grant, Boyd)

Matrix-Free Convex Optimization Modeling

(Diamond, Boyd)

CVX: http://cvxr.com/

CVXPY: http://www.cvxpy.org/

Convex.jl: http://convexjl.readthedocs.org/

Conclusions 44
http://cvxr.com/http://www.cvxpy.org/http://convexjl.readthedocs.org/http://convexjl.readthedocs.org/http://www.cvxpy.org/http://cvxr.com/

Documents

cvx_dcp