CSE 203B: Convex OptimizationWeek 3 Discuss Session

XinyuanWang01/31/2020

1

Contents

• Convex functions (Ref. Chap.3)• Definition• First order condition• Second order condition• Epigraph• Operationsthatpreserveconvexity

• Conjugatefunctions

2

Definition of Convex Functions• A function 𝑓: 𝑅$ → 𝑅 is convex if dom𝑓 is a convex set and if for all𝑥, 𝑦 ∈ dom𝑓 and 0 ≤ 𝜃 ≤ 1

𝑓 𝜃𝑥 + 1 − 𝜃 𝑦 ≤ 𝜃𝑓 𝑥 + 1− 𝜃 𝑓(𝑦)• Review the proof in class: necessary and sufficiency

• Strict convexity:𝑓 𝜃𝑥 + 1− 𝜃 𝑦 < 𝜃𝑓 𝑥 + 1 − 𝜃 𝑓 𝑦 , 𝑥 ≠𝑦, 0 < 𝜃 < 1• Concave functions:−𝑓 is convex

3

sometimes calledJensen’s inequality

(𝜃𝑥 + 1 − 𝜃 𝑦, 𝜽𝒇(𝒙) + 𝟏 −𝜽 𝒇(𝒚))

Example: convexity of functions with definition

4

i. Necessary: suppose𝑓 is convex, then 𝑓 satisfies the basic inequality𝑓 𝑥 + 𝜆 𝑦 − 𝑥 ≤ 𝑓 𝑥 + 𝜆(𝑓 𝑦 − 𝑓 𝑥 )

for0 ≤ 𝜆 ≤ 1. Integratingboth sides from0 to 1 on 𝜆

> 𝑓 𝑥 + 𝜆 𝑦 − 𝑥 𝑑𝜆@

A≤ > 𝑓 𝑥 + 𝜆 𝑓 𝑦 − 𝑓 𝑥 𝑑𝜆

@

A=𝑓 𝑥 + 𝑓(𝑦)

2ii. Sufficiency: suppose𝑓 is not convex, then there exists0 ≤ 𝜆D ≤ 1 and

𝑓 𝑥 + 𝜆D 𝑦 − 𝑥 > 𝑓 𝑥 + 𝜆D(𝑓 𝑦 − 𝑓 𝑥 )

there exist𝛼,𝛽 ∈ [0,1] so that in the interval [𝛼, 𝛽] the above

inequality alwaysholds,which contradicts the given inequality.𝜆D

𝛼𝛽

Restriction of a convex function to a line• A function 𝑓: 𝑅$ → 𝑅 is convex if and only if the function𝑔: 𝑅 → 𝑅,

𝑔 𝑡 = 𝑓 𝑥 + 𝑡𝑣 , dom𝑔 = 𝑡 𝑥 + 𝑡𝑣 ∈ dom𝑓}

is a convexon its domain for ∀𝑥 ∈ dom𝑓, 𝑣 ∈ 𝑅$.

• The property can be useful to check the convexity of a function

Example: Prove 𝑓 𝑋 = log det 𝑋 , dom𝑓 = 𝑆VV$ is concave.

5

Restriction of a convex function to a lineExample: Prove 𝑓 𝑋 = log det 𝑋 , dom𝑓 = 𝑆VV$ is concave.

Consider an arbitrary line𝑋 = 𝑍 + 𝑡𝑉, where 𝑍 ∈ 𝑆VV$ ,𝑉 ∈ 𝑆$. Define 𝑔 𝑡 =𝑓 𝑍 + 𝑡𝑉 and restrict 𝑔 to the interval values of 𝑡 for 𝑍 + 𝑡𝑉 ≻ 0.We have

𝑔 𝑡 = logdet(𝑍 + 𝑡𝑉) = logdet(𝑍@Z 𝐼 + 𝑡𝑍\

@Z𝑉𝑍\

@Z 𝑍@/Z)

= ^log 1 + 𝑡𝜆_

$

_`@

+ logdet 𝑍

where 𝜆_ are the eigenvalues of 𝑍\ab𝑉𝑍\

ab. So we have

𝑔c 𝑡 = ^𝜆_

1 + 𝑡𝜆_

$

_`@

, 𝑔cc 𝑡 = −^𝜆_Z

1 + 𝑡𝜆_ Z

$

_`@

≤ 0

𝑓 𝑋 is concave. For more practice, see Exercise 3.18.6

Ø The property of eigenvaluesdet𝐴 = ∏ 𝜆_$

_`@

First-order Condition• Suppose 𝑓 is differentiable (𝑑𝑜𝑚𝑓 is open and 𝛻𝑓 exists at ∀𝑥 ∈ 𝑑𝑜𝑚𝑓),then 𝑓 is convex iff dom𝑓 is convex and for all 𝑥, 𝑦 ∈ 𝑑𝑜𝑚𝑓

𝑓 𝑦 ≥ 𝑓 𝑥 + 𝛻𝑓 𝑥 j(𝑦− 𝑥)• Review the proof in class: necessary and sufficiency

• Strict convexity:𝑓 𝑦 > 𝑓 𝑥 + 𝛻𝑓 𝑥 j 𝑦− 𝑥 , 𝑥 ≠ 𝑦

• Concave functions: 𝑓 𝑦 ≤ 𝑓 𝑥 + 𝛻𝑓 𝑥 j(𝑦 − 𝑥)

7

a global underestimator

Proofoffirst-ordercondition:chap3.1.3withthepropertyofrestricting𝒇 toaline.

Second Order Condition• Suppose 𝑓 is twice differentiable (𝑑𝑜𝑚𝑓 is open and its Hessianexists at ∀𝑥 ∈ 𝑑𝑜𝑚𝑓), then 𝑓 is convex iff dom𝑓 is convexand for all𝑥, 𝑦 ∈ 𝑑𝑜𝑚𝑓

𝛻Z𝑓 𝑥 ≽ 0 (positive semidefinite)

• Review the proof in class: necessary and sufficiency

• Strict convexity:𝛻Z𝑓 𝑥 ≻ 0• Concave functions: 𝛻Z𝑓 𝑥 ≼ 0

8

Example of Convex Functions• Quadratic over linear function

𝑓 𝑥, 𝑦 =𝑥Z

𝑦 , for𝑦 > 0

Its gradient 𝛻𝑓 𝑥 =opoqopor

=

Zqr\qb

rb

Hessian 𝛻Z𝑓 𝑥 =obpoqb

obpoqor

obporoq

obporb

= Zrs

𝑦Z −𝑥𝑦−𝑥𝑦 𝑥Z

≽ 0 ⇒ convex

Positive semidefinite? 𝑦−𝑥

𝑦−𝑥

u, for any 𝑢 ∈ 𝑅Z, 𝑢j 𝑣𝑣j 𝑢 = 𝑣j𝑢 j 𝑣j𝑢 =

𝑣j𝑢 ZZ ≥ 0.

9Moreexamplesseechap.3.1.5

Epigraph• 𝛼-sublevel set of 𝑓: 𝑅$ → 𝑅

𝐶x = 𝑥 ∈ 𝑑𝑜𝑚𝑓 𝑓 𝑥 ≤ 𝛼}sublevel sets of a convex function are convex for any value of 𝛼.• Epigraph of 𝑓: 𝑅$ → 𝑅 is defined as

𝐞𝐩𝐢𝑓 = (𝑥, 𝑡) 𝑥 ∈ 𝑑𝑜𝑚𝑓, 𝑓 𝑥 ≤ 𝑡} ⊆ 𝑹𝒏V𝟏

• A function is convex iff its epigraph is a convex set. 10

Relation between convex sets and convex functions• A function is convex iff its epigraph is a convex set.

• Consider a convex function 𝑓 and 𝑥, 𝑦 ∈ 𝑑𝑜𝑚𝑓𝑡 ≥ 𝑓 𝑦 ≥ 𝑓 𝑥 + 𝛻𝑓 𝑥 j(𝑦 − 𝑥)

• The hyperplane supports epi 𝒇 at (𝑥, 𝑓 𝑥 ), for any

𝑦, 𝑡 ∈ 𝐞𝐩𝐢𝑓 ⇒𝛻𝑓 𝑥 j 𝑦− 𝑥 + 𝑓 𝑥 − 𝑡 ≤ 0

⇒ 𝛻𝑓 𝑥−1

j 𝑦𝑡 − 𝑥

𝑓 𝑥 ≤ 0

11

Supporting hyperplane, derivedfrom first order condition

First order condition for convexityepi 𝒇𝑡

𝑥

Operations that preserve convexityPractical methods for establishing convexity of a function• Verify definition ( often simplified by restricting to a line)• For twice differentiable functions, show 𝛻Z𝑓(𝑥) ≽ 0• Show that 𝑓 isobtained from simple convex functions by operationsthat preserve convexity (Ref. Chap. 3.2)• Nonnegativeweighted sum• Compositionwith affine function• Pointwisemaximum and supremum• Composition• Minimization• Perspective

12

Pointwise Supremum• If for each 𝑦 ∈ 𝑈, 𝑓(𝑥, 𝑦): 𝑅$ → 𝑅 is convex in 𝑥, then function

𝑔(𝑥) = supr∈�

𝑓(𝑥, 𝑦)

is convex in 𝑥.• Epigraph of 𝑔 𝑥 is the intersection of epigraphs with 𝑓 and set 𝑈

𝐞𝐩𝐢𝑔 =∩r∈� 𝐞𝐩𝐢𝑓(�, 𝑦)knowing 𝐞𝐩𝐢𝑓(�, 𝑦) is a convex set (𝑓 is a convex function in 𝑥 and regard

𝑦 as a const), so 𝐞𝐩𝐢𝑔 is convex.

• An interesting method to establish convexity of a function: the pointwisesupremum of a family of affine functions. (ref. Chap 3.2.4)

13

Contents

• Convex functions (Ref. Chap.3)• Definition• First order condition• Second order condition• Epigraph• Operationsthatpreserveconvexity

• Conjugatefunctions

14

Conjugate Function• Given function 𝑓: 𝑅$ → 𝑅, the conjugate function

𝑓∗ 𝑦 = supq∈���p

𝑦j𝑥 − 𝑓(𝑥)

• The dom 𝑓∗ consists 𝑦 ∈ 𝑅$ for which supq∈���p

𝑦j𝑥 − 𝑓(𝑥) is bounded

Theorem: 𝑓∗(𝑦) is convex even 𝑓 𝑥 is not convex.15

(Proof with pointwise supremum)𝒚𝑻𝒙 − 𝒇(𝒙) is affine function in 𝒚

Supportinghyperplane: if𝑓 is convex in that domain

(𝑥� , 𝑓(𝑥�)) where𝛻𝑓j 𝑥� = 𝑦 if 𝑓 is differentiable

(0,−𝑓∗(0))

Slope 𝑦 = −1

Slope 𝑦 = 0

Examples of Conjugates• Derive the conjugates of 𝑓: 𝑅 → 𝑅

𝑓∗ 𝑦 = supq∈���p

𝑦𝑥 − 𝑓(𝑥)

16

Affine

Normquadratic

See moreexamplesinchap3.3.1

Properties of Conjugates

17

(1)𝑓 𝑥 + 𝑓∗ 𝑦 ≥ 𝑥j𝑦Fenchel’sinequality.Thus,intheaboveexample

𝑥j𝑦 ≤ @Z𝑥j𝑄𝑥 + @

Z𝑦j𝑄\@𝑦, ∀𝑥, 𝑦 ∈ 𝑅$,𝑄 ∈ 𝑆VV$

(2)𝑓∗∗ = 𝑓,iff isconvex&f isclosed(i.e.epif isaclosedset)(3)Iff isconvex&differentiable,𝑑𝑜𝑚𝑓 = 𝑅$

Formax𝑦j𝑥 − 𝑓(𝑥),wehave𝑦 = 𝛻𝑓(𝑥∗)Thus,𝑓∗ 𝑦 = 𝑥∗j𝛻𝑓 𝑥∗ − 𝑓 𝑥∗ , 𝑦 = 𝛻𝑓(𝑥∗)

ConjugateFunction

18

• Aneconomicinterpretation𝑓 𝑥 :costtoproduceproductwithquantity𝑥𝑦:marketprice𝑓∗(𝑦):optimalprofitatgivenprice𝑦

• ConjugatefunctionandtheLagrangedualfunction(willbecoveredinlaterclass)• Optimizationproblemwithlinearconstraints