An Introduction to CVX - KTHamishi/PDF/Presentation_Nov_2014.pdf · WhatisCVX? • CVXis a MATLAB–based software package for solving convex optimization problems. • Feature: CVXis

An Introduction to CVX

Amirpasha Shirazinia

Signals and Systems Division

Department of Engineering Sciences

Uppsala University

November 4, 2014

Amirpasha Shirazinia (UU) An Introduction to CVX Internal Seminar 1 / 17

What is CVX ?

• CVX is a MATLAB–based software package for solving convexoptimization problems.

• Feature: CVX is written in a high–level form, i.e., close tomathematical notations.

• 1748 citations (according to google).


What is CVX ?




• Restriction: can be applied to convex programs,

• What if the problem you are dealing with is non-convex:• Try to convexify the non-convex problem,• Try to relax the problem in order to make it convex,• Go for a sub-optimal locally minimizing approach, e.g., fmincon.


What is CVX ?




• Restriction: can be applied to convex programs,

• What if the problem you are dealing with is non-convex:• Try to convexify the non-convex problem,• Try to relax the problem in order to make it convex,• Go for a sub-optimal locally minimizing approach, e.g., fmincon.

• Download & install CVX .


http://cvxr.com/cvx/download/

Scalar Example

minimize (x+ y + 3)2

subject to y ≥ 00 ≤ x ≤ 1.

(1)

How to translate Problem (1) into CVX ?


Scalar Example



(1)

How to translate Problem (1) into CVX ?>> cvx begin


Scalar Example



(1)


>> variables x y


Scalar Example



(1)


>> variables x y

>> y >= 0


Scalar Example



(1)


>> variables x y

>> y >= 0

>> x >= 0


Scalar Example



(1)


>> variables x y

>> y >= 0

>> x >= 0

>> x <= 1


Scalar Example



(1)


>> variables x y

>> y >= 0

>> x >= 0

>> x <= 1

>> minimize((x+ y+ 3)2)


Scalar Example



(1)


>> variables x y

>> y >= 0

>> x >= 0

>> x <= 1

>> minimize((x+ y+ 3)2)>> cvx end


Some Useful Commands & Hints

• cvx status: gives the status of the solution –solved/infeasible/unbounded,

• cvx optval: gives the optimal value,

• cvx quiet(true): displays off the optimization procedure,

• Affine equality constraints are denoted by ==,

• Flexibility: CVX commands can be used in MATLAB loops,

• Lots of other hints available in CVX user’s guide.


http://cvxr.com/cvx/doc/

Vector Example

• Problem 4.8e (Boyd): Finding optimal probabilities for some weightingcoefficients {αi}Ni=1

minimize∑N

i=1αipi

subject to pi ≥ 0 , ∀i∑N

i=1pi = 1.

(2)


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)

>> N = 10; alpha = rand(1, N)


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)

>> N = 10; alpha = rand(1, N)>> cvx begin


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)


>> variable p(N)


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)


>> variable p(N)>> p >= 0


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)



>> sum(p) == 1


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)



>> sum(p) == 1

>> minimize(alpha ∗ p)


Vector Example


minimize∑N

i=1αipi


i=1pi = 1.

(2)



>> sum(p) == 1

>> minimize(alpha ∗ p)>> cvx end


Vector Example

• Problem 4.26 (Boyd):

minimize x⊤x+ y + zsubject to x⊤x ≤ yz

y ≥ 0 , z ≥ 0(3)

• Question 1: Is the objective function convex?


Vector Example



y ≥ 0 , z ≥ 0(3)

• Question 1: Is the objective function convex? Yes! ,

• Question 2: Are the constraints convex sets?


Vector Example



y ≥ 0 , z ≥ 0(3)


• Question 2: Are the constraints convex sets? Let’s call CVX first �

>> N = 10;>> cvx begin

>> variables x(N, 1) y z

>> y >= 0

>> z >= 0

>> x′ ∗ x− y ∗ z <= 0


Vector Example



y ≥ 0 , z ≥ 0(3)


• Question 2: Are the constraints convex sets? Let’s call CVX first �

>> N = 10;>> cvx begin


>> y >= 0

>> z >= 0

>> x′ ∗ x− y ∗ z <= 0

Disciplined convex programming error:


Vector Example (Cont’d)

• Why the constraint x⊤x ≤ yz does not form a convex region?




• convex ≤ ???




• convex ≤ ???• ??? should be an affine or concave function.




• convex ≤ ???• ??? should be an affine or concave function.• Now, via the Hessian matrix, check if f(y, z) = yz is concave (Problem

3.16 Boyd).





3.16 Boyd).

• How to convexify Problem (2)?

x⊤x ≤ yz ⇐⇒

∥∥∥∥[

2xy − z

]∥∥∥∥2

≤ y + z. (4)





3.16 Boyd).

• How to convexify Problem (2)?

x⊤x ≤ yz ⇐⇒

∥∥∥∥[

2xy − z

]∥∥∥∥2

≤ y + z. (4)

• Now, we have a convex problem ,

minimize x⊤x+ y + z

subject to

∥∥∥∥[

2xy − z

]∥∥∥∥2

≤ y + z

y ≥ 0 , z ≥ 0

(5)



• CVX code:>> N = 10;>> cvx begin


>> y >= 0

>> z >= 0

>> norm([2x; y− z], 2) <= y+ z

>> minimize(x′ ∗ x+ y+ z)>> cvx end


Matrix Example: Sensing/Measurement Matrix design in a

Bayesian Framework

• Consider the linear Gaussian model y = Ax+ n,

• where x ∼ N (0,R), n ∼ N (0, σ2I), such that x,y,n ∈ RN and

A ∈ RN×N .

• The MMSE estimator of x given the noisy measurements y becomes

xmmse = arg minx

E[‖x− x‖22] =

1

σ2

(R−1 +

1

σ2A⊤A

)−1

A⊤y

• The corresponding MSE, to be minimized, is

MSE = Tr

{(R−1 +

1

σ2A⊤A

)−1}

(6)


Matrix Example: Sensing/Measurement Matrix design in a

Bayesian Framework

• Consider the linear Gaussian model y = Ax+ n,

• where x ∼ N (0,R), n ∼ N (0, σ2I), such that x,y,n ∈ RN and

A ∈ RN×N .

• The MMSE estimator of x given the noisy measurements y becomes

xmmse = arg minx

E[‖x− x‖22] =

1

σ2

(R−1 +

1

σ2A⊤A

)−1

A⊤y

• The corresponding MSE, to be minimized, is

MSE = Tr

{(R−1 +

1

σ2A⊤A

)−1}

(6)

• Assume the total available power P > 0, we have

E[‖Ax‖22] = . . . = Tr

{ARA⊤

}≤ P (7)


Matrix Example (Cont’d)

• The sensing/measurement matrix design problem is posed as

minimize Tr{(

R−1 + 1

σ2A⊤A

)−1}

subject to Tr{RA⊤A

}≤ P

(8)



• The sensing/measurement matrix design problem is posed as

minimize Tr{(

R−1 + 1

σ2A⊤A

)−1}

subject to Tr{RA⊤A

}≤ P

(8)

• Problem (8) is not convex in A, but convex in A⊤A.

• Let X , A⊤A, we have an equivalent optimization problem

minimize Tr{(

R−1 + 1

σ2X)−1

}

subject to Tr {RX} ≤ PX � 0

(9)



• Let’s translate the optimization problem (8) into CVX codes>> N = 3; rho = 0.5; P = 10; sigma sq = 0.01;>> R = [1 rho rho 2; rho 1 rho ; rho 2 rho 1];>> cvx begin sdp

>> variable X(N, N) symmetric>> X == semidefinite(N, N)>> trace(R ∗ X) <= P

>> minimize(trace inv(R (−1) + X/sigma sq))



• Let’s translate the optimization problem (8) into CVX codes>> N = 3; rho = 0.5; P = 10; sigma sq = 0.01;>> R = [1 rho rho 2; rho 1 rho ; rho 2 rho 1];>> cvx begin sdp

>> variable X(N, N) symmetric>> X == semidefinite(N, N)>> trace(R ∗ X) <= P

>> minimize(trace inv(R (−1) + X/sigma sq))>> cvx end

• How to get back A from X?• For example, use the cholesky factorization; in MATLAB A = chol(X).


A Little into Compressed Sensing (CS)

• CS answers the following question:• In a linear set of equations y = Ax, would it be possible to uniquely

solve the equations if the number of unknowns, i.e. dim(x), is largerthan the number of equations, i.e. dim(y)?


A Little into Compressed Sensing (CS)

• CS answers the following question:• In a linear set of equations y = Ax, would it be possible to uniquely

solve the equations if the number of unknowns, i.e. dim(x), is largerthan the number of equations, i.e. dim(y)?

• Yes! But under the condition that the unknown variables are knownto be sparse (K ≪ N) plus some mild conditions.

y

x

=

A

M N

K


Optimization framework for CS

• We know that the source, x, is sparse.




• We observe the linear measurements y = Ax.





• Goal: Go for the sparsest solution.

minimize ‖x‖0subject to y = Ax

(10)

• where the ℓ0–norm ‖x‖0 , card(x) = #non-zero components.

• Check whether or not the optimization problem (10) is convex ...• Hint: Check the geometry of ℓ0 ...





• Goal: Go for the sparsest solution.


(10)

• where the ℓ0–norm ‖x‖0 , card(x) = #non-zero components.

• Check whether or not the optimization problem (10) is convex ...• Hint: Check the geometry of ℓ0 ...• NOT CONVEX!


Optimization framework for CS (Cont’d)

• How to tackle the problem?

• Standard approach: Relax the objective function to the nearestconvex norm, i.e., ℓ1–norm, a.k.a. basis pursuit (BP),


(11)

• where ‖x‖1 =∑

N

n=1|xn|.


Optimization framework for CS (Cont’d)

• How to tackle the problem?

• Standard approach: Relax the objective function to the nearestconvex norm, i.e., ℓ1–norm, a.k.a. basis pursuit (BP),


(11)

• where ‖x‖1 =∑

N

n=1|xn|.

• The miracle happens: Under some conditions, the solutions to (11)and (10) are the same. CS theory also determines how manymeasurements need to be acquired in order for this equivalence tohold.


Optimization framework for CS: Example

>> N = 50; M = 25; K = 5;>> supp = randsample(N, K)′; x = zeros(N, 1); x(supp) = randn(K, 1);>> A = randn(M, N); s = sqrt(sum(A.2)); S = diag(1./s);>> A = A ∗ S;>> y = A ∗ x;>> cvx begin

>> variable x hat(N, 1)>> y == A ∗ x hat

>> minimize(norm(x hat, 1))>> cvx end

>> plot(1 : N, x, 1 : N,′ s′, x hat,′ o′)


Conclusion

• CVX is cool!


References

• S. Boyd and L. Vandenberghe, Convex optimization, CambridgeUniversity Press, 2004.

• http://cvxr.com/cvx.

• A. Shirazinia, Source and channel coding for compressed sensing and

control, PhD thesis, KTH, 2014.


http://cvxr.com/cvx

Documents

An Introduction to CVX - KTHamishi/PDF/Presentation_Nov_2014.pdf · WhatisCVX? • CVXis a MATLAB–based software package for solving convex optimization problems. • Feature: CVXis