L11 Optimal Design L.Multipliers• Homework• Review• Convex sets, functions• Convex Programming Problem• Summary

1

Constrained OptimizationLaGrange Multiplier Method

2

nkx x x

mjg

pi= h

f :MINIMIZE

(L )kk

(U )k

j

i

1

...10)(

...10)(: ToSubject

) (

x

x

x

Remember:1.Standard form2.Max problems f(x) = - F(x)

KKT Necessary Conditions for Min

3

))(( )()( 2

11i

m

iii

p

iii sguhυfL

xxxsu,v,x,

nkx

gu

x

hυ

x

f m

j k

jj

p

i k

ii

k

to1for 011

mjsg

ph

jj

i

to1for 0*)(

to1ifor 0*)(2

x

x

Regularity check - gradients of active inequality constraints are linearly independent

mjs j to1for 02 mjsu jj to1for 0*

mju j to1for 0*

Prob 4.120

4

02

04subject to

)1()1(),(

12

211

22

2121

xg

xxg

xxxxfMin

]201[]4[

...)1()1(

)()(),(

22212

21211

22

21

2222

211121

sxxusxxu

xxL

sgusguxxfL

KKT Necessary Conditions

5

int?Regular po00

0,0

0,0

0201

0411

00)1(2

0)1(2

:ConditionsNecessary KKT

21

2211

22

21

22212

21211

122

2111

uu

susu

ss

sxxg

sxxg

uxx

L

uuxx

L

Case 1

6

!012)1(0)1(1

02...04)1(1)1(1

0201

0411

1,1000)1(2000)1(2

0,01Case

22

21

21

22212

21211

21

2

1

21

BADs

ss

sxxg

sxxg

xxxx

uu

0,4 Case0,3 Case0,2 Case0,1 Case

21

21

21

21

21211

21211

22

ssussuuu

ssuss

suuuu

su

Case 2

7

1

04)1(1)2(1

2,1000)1(2

00201

0411

000)1(200)1(2

0,0 2. Case

21

21

12

2

21

2121

2

21

21

s

s

xxx

xx

sxx

xux

su

1,2,0),1,2( :pointKKT

0,0,0,0:

200)12(2

21

2122

21

2

2

fuux

uusscheck

uu

Regularity: 1. pt is feasible2. only one active constraintPoint is KKT pt, OK!

Case 3

8

4011

21202102

00)1(200)1(20,0Case3.

121

121

121

12

11

21

uxx

uxxuxx

uxux

us

2

!,242

6120

21202102

2,1

1

1

121

121

121

x

BADuu

uxx

uxxuxx

Case 4

9

!0,2

2,2

00201

00411

2020202

00)1(20)1(2

0,s 4. Case

21

21

21

21

2121

2121

12

211

21

BADuu

xx

xx

xx

uuxxuuxx

uxuux

s

Sensitivity or Case 2

10

12,0

)1,2(:pointKKT2

21

fuu

xCase

0)1,0(1)1(21)0.1,0(%50

68.0)2.0,0(6.0)2.0(21)2.0,0(%10

)0,0(*),()0,0(*),(

2211

factualffactual

feueufebfeubυfebf

ji

jjiiji

Constraint Variation Sensitivity

From convexity theorems:1. Constraints linear2. Hf is PDTherefore KKT Pt is global Min!

Graphical Solution

11

3 4

21

00

02

00

02

00

02

00

)11(2)12(2

00

01

211

0)1(2)1(2

0

2

1

2211

xx

guguf

LaGrange Multiplier Method• May produce a KKT point• A KKT point is a CANDIDATE minimum

It may not be a local Min

• If a point fails KKT conditions, we cannot guarantee anything….The point may still be a minimum.

• We need a SUFFICIENT condition12

Recall Unconstrained MVO

13

0x *)( Tf

For x* to be a local minimum:

2

1*)( dHddx TTff

0 *)()( xx fff

1rst orderNecessaryCondition

0 dHdT

2nd orderSufficientCondition

i.e. H(x*) must be positive definite

Considerationsfor Constrained Min?

14

Objective functionDifferentiable, continuous i.e. smooth?Hf(x) Positive definite (i.e. convexity of f(x) )

Weierstrass theorem hints:x closed and boundedx contiguous or separated, pockets of points?

Constraints h(x) & g(x) Define the constraint set, i.e. feasible regionx contiguous or separated, pockets of points?

Convex: sets, functions, constraint set and Programming problem

Punchline (Theorem 4.10, pg 165)

15

The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:

1. f(x) is convexHf(x) Positive definite

2. x is defined as a convex feasible set.Equality constraints must be linearInequality constraints must be convex

HINT: linear functions are convex!

16

Convex set:All pts in feasible region on a straight line(s).

Convex sets

Non-convex setPts on line are not in feasible region

17

10)(10;)1(

121

12

αxxαxxαxααxx

Single variable

No “gaps” in feasible “region”

18

10);(10;)1(

)1()2()2(

)2()2(

ααααα

xxxxxxx

Multiple variables Fig 4.21

0122

21 xx

What if it were an equality constraint?

misprint

19

.

Figure 4.22 Convex function f(x)=x2

Bowl that holds water.

10)]()([)())1((10))()(()()(

10;)()1()()(

12112

121

12

αxfxfαxfxααxfαxfxfαxfxf

αxfαxαfxf

2010)]()([)())1((

10))()(()()(10;)()1()()(

12112

121

12

αxfxfαxfxααxfαxfxfαxfxf

αxfαxαfxf

Fig 4.23 Characterization of a convex function.

Test for Convex Function

21

10))()(()()(

10;)()1()()()1()2()1(

)1()2(

αffαff

αfααff

xxxx

xxx

Difficult to use above definition!

However, Thm 4.8 pg 163:If the Hessian matrix of the function is PD ro PSD at all points in the set S, then it is convex.

PD… “strictly” convex, otherwisePSD… “convex”

Theorem 4.9

22

}to1,0)(g; to1for,0)(|{

SetConstraint

j mjpihS i

xxx

Given:

S is convex if:1. hi are linear2. gj are convex i.e. Hg PD or PSD

When f(x) and S are convex= “convex programming problem”

“Sufficient” Theorem 4.10, pg 165

23

The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:

1. f(x) is convexHf(x) Positive definite

2. x is defined as a convex feasible set SEquality constraints must be linearInequality constraints must be convex

HINT: linear functions are convex!

Summary• LaGrange multipliers are the

instantaneous rate of change in f(x) w.r.t. relaxing a constraint.

• KKT point is a CANDIDATE min!(need sufficient conditions for proof)

• Convex sets assure contiguity and or the smoothness of f(x)

• KKT pt of a convex progamming problem is a GLOBAL MINIMUM!

24