Osc Slides Opt Intro

8/8/2019 Osc Slides Opt Intro

1/30

Optimization Introduction

Optimization in Systems and Control Optimization Introduction 1 / 30


2/30

Optimization deals with how to do things in the best possiblemanner:

Design of multi-criteria controllers Clustering in fuzzy modeling Trajectory planning of robots Scheduling in process industry Estimation of system parameters Simulation of continuous time systems on digital computers Design of predictive controllers with input-saturation

Related courses:

SC4060: Model predictive control SC4040: Filtering & identication WI4217: Optimal control theory: Fundamentals & related

topics



3/30

Overview

Three subproblems:Formulation:

Translation of engineering demands and requirementsinto a mathematical well-dened optimizationproblem

Initialization: Choice of right algorithmChoice of initial values for parameters

Optimization procedure;Various optimization techniquesVarious computer platforms



4/30

Teaching goals

I. Optimization problem most efficient and best suitedoptimization algorithm

II. Engineering problem optimization problem: specications mathematical formulation simplications/approximations? efficiency implementation



5/30

Contents

Part I. Optimization TechniquesPart II. Formulating the Controller Design Problem as an

Optimization Problem



6/30

Contents of Part I

I. Optimization Techniques1. Introduction2. Linear Programming3. Quadratic Programming4. Nonlinear Optimization5. Constraints in Nonlinear Optimization6. Convex Optimization7. Global Optimization8. Summary9. Matlab Optimization Toolbox

10. Multi-Objective OptimizationAppendix E. Integer Optimization



7/30

Mathematical framework

minx

f (x )

s.t. h(x ) = 0g (x ) 0

f : objective function x : parameter vector h(x ) = 0 : equality constraints

g (x ) 0 : inequality constraintsf (x ) is a scalarg (x ) and h(x ) may be vectors



8/30

Unconstrained optimization:

f (x

) = minx f (x )

wherex = arg min

x f (x )

Constrained optimization:

f (x ) = minx f (x )

h(x ) = 0

g (x

) 0where

x = arg minx

f (x ), h(x ) = 0 , g (x ) 0



9/30

Maximization = Minimization

maxx

f (x ) = minx

f (x )

f

f

x



10/30

Classes of optimization problems

Linear programmingmin

x c T x , A x = b , x 0

minx

c T x , A x b , x 0 Quadratic programming

minx

x T Hx + c T x , A x = b , x 0

minx

x T Hx + c T x , A x b , x 0 Convex optimization

minx

f (x ) , g (x ) 0 where f and g are convex

Nonlinear optimization

minx

f (x ) , h(x ) = 0 , g (x ) 0

where f , h , and g are non-convex and nonlinearOptimization in Systems and Control Optimization Introduction 10 / 30


11/30

Convex set

Set C in Rn

is convex if x , y C, [0, 1] :(1 )x + y C

x ( = 0)

y ( = 1)

(1 )x + y



12/30

Unimodal function

A function f is unimodal if a) The domain dom( f ) is a convex set.b) x dom(f ) such

f (x ) f (x ) x dom(f )

c) For all x 0 dom(f )

there is a trajectory x ( ) dom(f )

with x (0) = x 0 and x (1) = x

such that

f x ( ) f (x 0 ) [0, 1]



13/30

Inverted Mexican hat

f (x ) = x T x

1 + x T x x R 2

42

02

4

4

2

0

2

4

0

0.2

0.4

0.6

0.8

1

x 1x 2

f

4 3 2 1 0 1 2 3 44

3

2

1

0

1

2

3

4

x 1

x 2



14/30

Rosenbrock function

f (x 1 , x 2 ) = 100( x 2 x 21 )2 + (1 x 1 )2

2

1

0

1

2

1

0

1

2

30

500

1000

1500

2000

2500

3000

x 1x 2

f

2 1.5 1 0.5 0 0.5 1 1.5 21

0.5

0

0.5

1

1.5

2

2.5

3

x 1

x 2



15/30

Quasiconvex function

A function f is quasiconvex if

a) Domain dom(f ) is a convex setb) For all x , y dom(f )

and 0 1

there holds

f (1 )x + y max(f (x ), f (y ))



16/30

Quasiconvex function

Alternative denition:A function f is quasiconvex if the sublevel set

L( ) = { x dom(f ) : f (x ) }is convex for every real number



17/30

Convex function

A function f is convex if

a) Domain dom(f ) is a convex set.b) For all x , y dom(f )

and 0 1

there holds

f (1 )x + y (1 )f (x ) + f (y )



18/30

Convex function

x

( = 0)

y

( = 1)

(1 )x + y

f (1 )x + y

(1 )f (x ) + f (y )

f (x )

f (y )



19/30

Test: Unimodal, quasiconvex, convex

Given: Function f with graphf

Question: f is (check all that apply)unimodalquasiconvex

convexOptimization in Systems and Control Optimization Introduction 19 / 30


20/30

Gradient and Hessian

Gradient of f : f (x ) =

f x 1 f

x 2... f x n

Hessian of f : H (x ) =

2 f x 21

2 f x 1 x 2

. . . 2 f

x 1 x n

2 f

x 1 x 2

2 f

x 22

. . . 2 f

x 2 x n...... . . .

... 2 f

x 1 x n 2 f

x 2 x n. . .

2 f x 2n



21/30

Jacobian

x : vectorh: vector-valued

h(x ) =

h 1 x 1

h 2 x 1

. . . hm x 1

h 1 x 2 h 2 x 2 . . . h

m x 2

...... . . .

... h 1

x n

h 2

x n. . . hm

x n



22/30

Graphical interpretation of gradient

Directional derivative of function f in x 0 in direction of unitvector :

D f (x 0 ) = T f (x 0 ) = f (x 0 ) 2 cos

with angle between f (x 0 ) and

D f (x 0 ) is maximal if f (x 0 ) and are parallel function values exhibit largest increase in direction of f (x 0 ) function values exhibit largest decrease in direction of f (x 0 )

f (x 0 ) is called steepest descent direction

D f (x 0 ) is equal to 0 (i.e., function values f do not change) if f (x 0 )

f (x 0 ) is perpendicular to contour line through x 0



23/30

f

x 0

f (x 0 )

x 1

x 2

f (x 0 )

tan = D f (x 0 )



24/30

Subgradient

Let f be a convex function. f (x 0 ) is a subgradient of f in x 0 if

f (x ) f (x 0 ) + T f (x 0 ) (x x 0 )

for all x R n

f

f (x 0 ) + T f (x 0 )(x x 0 )

x 0x 0Optimization in Systems and Control Optimization Introduction 24 / 30


25/30

Classes of optimization problems

Linear programmingmin

x c T x , A x = b , x 0

minx

c T x , A x b , x 0 Quadratic programming

minx

x T Hx + c T x , A x = b , x 0

minx x T Hx + c T x , A x b , x 0 Convex optimization

minx

f (x ) , g (x ) 0 where f and g are convex

Nonlinear optimization

minx

f (x ) , h(x ) = 0 , g (x ) 0

where f , h , and g are non-convex and nonlinearOptimization in Systems and Control Optimization Introduction 25 / 30


26/30

Conditions for extremum learn by heart!

Unconstrained optimization problem: minx f (x )Zero-gradient condition : f (x ) = 0

Equality constrained optimization problem: minx

f (x )

s.t. h(x ) = 0Lagrange conditions: f (x ) + h(x ) = 0

h(x ) = 0

Inequality constrained optimization problem: minx f (x )s.t. g (x ) 0

h(x ) = 0

Kuhn-Tucker conditions :

f (x ) + g (x ) + h(x ) = 0

T g (x ) = 0 0

h(x ) = 0

g (x ) 0Optimization in Systems and Control Optimization Introduction 26 / 30


27/30

Convergence

1 = limk

x k +1 x

x k x

Linear convergent if 0< 1 < 1

Super-linear convergent if 1 = 0

2 = limk

x k +1 x

x k x 2

Quadratic convergent if 0 < 2 < 1


S i i i


28/30

Stopping criteria Linear and Quadratic programming: Finite number of steps Convex optimization: f (x k ) f (x ) f , g (x k ) g , and

for ellipsoid: x k x x Unconstrained nonlinear optimization: f (x k ) Constrained nonlinear optimization:

f (x k ) + g (x k ) + h(x k ) KT 1

T

g (x k ) KT 2 KT 3

h(x k ) KT 4

g (x k ) KT 5

Maximum number of steps Heuristic stopping criteria (last resort):

x k +1 x k x or f (x k ) f (x k +1 ) f Optimization in Systems and Control Optimization Introduction 28 / 30

S


29/30

Summary

Standard form of optimization problem:min

x f (x ) s.t. h(x ) = 0 , g (x ) 0

Classes of optimization problems: linear, quadratic, convex,nonlinear

Convex sets & functions Gradient, subgradient, and Hessian Conditions for extremum Stopping criteria


T t G di t


30/30

Test: Gradient

Given: Level lines of unimodal function f with minimum x

, apoint x 0 , and vectors v 1 , v 2 , v 3 , v 4 , v 5 , one of which is equal to f (x 0).

Question: Which vector v i

is equal to f (x 0)?

x

x 0

v 1v 2

v 3v 4 v 5


Documents

Osc Slides Opt Intro