Upload
tranque
View
227
Download
0
Embed Size (px)
Citation preview
Class Exercises From Last Time
2 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Today’s Material
• Extrema
• Convex Function
• Convex Sets
• Other Convexity Concepts
• Unconstrained Optimization
3 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Extrema
Problem
max f(x) s.t. x ∈ S
max {f(x) : x ∈ S}
• Global maximum x∗
f(x∗) ≥ f(x) ∀x ∈ S
• Local maximum xo:
f(xo) ≥ f(x) ∀x in a neighborhood around xo
• strict maximum/minimum defined similarly
4 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Weierstrass theorem:
Theorem
A continuous function achieves its max/min on a closed and bounded set
5 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Supremum and Infimum
Supremum
The supremum of a set S having a partial order is the least upper bound of S(if it exists) and is denoted sup S.
Infimum
The infimum of a set S having a partial order is the greatest lower bound of S(if it exists) and is denoted inf S.
• If the extrema are not achieved:
• max → sup
• min → inf
• Examples
• sup{2, 3, 4, 5}?• sup{x ∈ Q : x2 < 2}?• inf{1/x : x > 0}?
6 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Finding Optimal Solutions
Every method for finding and characterizing optimal solutions is based onoptimality conditions - either necessary or sufficient
Necessary Condition
A condition C1(x) is necessary if C1(x∗) is satisfied by every optimal solution
x∗ (and possibly some other solutions as well).
Sufficient Condition
A condition C2(x) is sufficient if C2(x∗) ensures that x∗ is optimal (but some
optimal solutions may not satisfy C2(x∗).
Mathematically
{x|C2(x)} ⊆ {x|x optimal solution } ⊆ {x|C1(x)}
7 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Finding Optimal Solutions
An example of a necessary condition in the case S is “well-behaved” noimproving feasible direction
Feasible Direction
Consider xo ∈ S, s ∈ Rn is called a feasible direction if there exists ε(s) > 0 suchthat
xo + εs ∈ S ∀ε : 0 < ε ≤ ε(s)
We denote the cone of feasible directions from xo in S as S(xo)
Improving Direction
s ∈ Rn is called an improving direction if there exists ε(s) > 0 such that
f(xo + εs) < f(xo) ∀ε : 0 < ε ≤ ε(s)
The cone of improving directions from xo in S is denoted F (xo)
8 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Finding Optimal Solutions
Local Optima
If xo is a local minimum, then there exist no s ∈ S(xo) for which f(.) decreasesalong s, i.e. for which
f(xo + ε2s) < f(xo + ε1s) for 0 ≤ ε1 < ε2 ≤ ε(s)
Stated otherwise: A necessary condition for local optimality is
F (xo) ∩ S(xo) = ∅
9 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Improving Feasible Directions
If for a given direction s it holds that
∇f(xo)s < 0
Then s is an improving direction
Well known necessary condition for local optimality of xo for adifferentiable function:
∇f(xo) = 0
In other words, xo is stationary with respect to f(.)
10 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
What if stationarity is not enough?
• Suppose f is twice continuously differentiable
• Analyse the Hessian matrix for f at xo
∇2f(xo) =
{∂2(f(xo))
∂xi∂xj
}, i, j = 1, . . . , n
Sufficient Condition
If ∇f(xo) = 0 and ∇2f(xo) is positive definite:
xT∇2f(xo)x > 0 ∀ x ∈ Rn\{0}
then xo is a local minimum
11 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
What if Stationarity is not Enough?
• A necessary condition for local optimality is “Stationarity + positivesemidefiniteness of ∇2f(xo)”
• Note that positive definiteness is not a necessary condition
• E.g. look at f(x) = x4 for xo = 0
• Similar statements hold for maximization problems
• Key concept here is negative definiteness
12 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Definiteness of a Matrix
• A number of criteria regarding the definiteness of a matrix exist
• A symmetric n× n matrix A is positive definite if and only if
xTAx > 0 ∀x ∈ Rn\{0}
• Positive semidefinite is defined likewise with ′′ ≥′′ instead of ′′ >′′
• Negative (semi) definite is defined by reversing the inequality signs to ′′ <′′ and′′ ≤′′, respectively.
Necessary conditions for positive definiteness:
• A is regular with det(A) > 0
• A−1 is positive definite
13 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Definiteness of a Matrix
Necessary+Sufficient conditions for positive definiteness:
• Sylvestor’s Criterion: All principal submatrices have positive determinants
(a11)
(a11 a12
a21 a22
) a11 a12 a13
a21 a22 a23
a31 a32 a33
• All eigenvalues of A are positive
14 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Necessary and Sufficient Conditions
Theorem
Suppose that f(.) is differentiable at a local minimum xo. Then ∇f(xo)s ≥ 0 fors ∈ S(xo). If f(.) is twice differentiable at xo and ∇f(xo) = 0, thensT∇2f(xo)s ≥ 0 ∀s ∈ S(xo)
Theorem
Suppose that S is convex and non-empty, f(.) differentiable, and that xo ∈ S.Suppose furthermore that f(.) is convex.
• xo is a local minimum if and only if xo is a global minimum
• xo is a local (and hence global) minimum if and only if
∇f(xo)(x− xo) ≥ 0 ∀x ∈ S
15 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Convex Combination
Convex combination
The convex combination of two points is the line segment between them
α1x1 + α2x2 for α1, α2 ≥ 0 and α1 + α2 = 1
16 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Convex function
Convex Functions
A convex function lies below its chord
f(α1x1 + α2x2) ≤ α1f(x1) + α2f(x2)
• A strictly convex function has no more than one minimum
• Examples: y = x2, y = x4, y = x
• The sum of convex functions is also convex
• A differentiable convex function lies above its tangent
• A differentiable function is convex if its Hessian is positive semi-definite
• Strictly convex not analogous!
• A function f is concave iff −f is convex
EOQ objective function: T (Q) = dK/Q+ cd+ hQ/2?
17 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Convex function
Convex Functions
A convex function lies below its chord
f(α1x1 + α2x2) ≤ α1f(x1) + α2f(x2)
• A strictly convex function has no more than one minimum
• Examples: y = x2, y = x4, y = x
• The sum of convex functions is also convex
• A differentiable convex function lies above its tangent
• A differentiable function is convex if its Hessian is positive semi-definite
• Strictly convex not analogous!
• A function f is concave iff −f is convex
EOQ objective function: T (Q) = dK/Q+ cd+ hQ/2?
17 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Economic Order Quantity Model
The problem
The Economic Order Quantity Model is an inventory model that helpsmanafacturers, retailers, and wholesalers determine how they should optimallyreplenish their stock levels.
Costs
• K = Setup cost for odering one batch
• c = unit cost for producing/purchasing
• h = holding cost per unit per unit of time in inventory
Assumptions
• d = A known constant demand rate
• Q = The order quantity (arrives all at once)
• Planned shortages are not allowed
18 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Convex sets
Definition
A convex set contains all convex combinations of its elements
α1x1 + α2x2 ∈ S ∀ x1,x2 ∈ S
• Some examples of E.g. (1, 2], x2 + y2 < 4, ∅• Level curve (2 dimensions):
{(x, y) : f(x, y) = β}
• Level set:{x : f(x) ≤ β}
19 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Lower Level Set Example
x2
x1
x1 + x2 ≤ 3
x1 + 3x2 ≤ 7
20 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Lower Level Set Example
y2
y1
y1 + y2 ≥ 1y1 + 3y2 ≥ 2
21 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Upper Level Set Example
−4−2
02
4 −4−2
02
4
0
100
x y
zPlot of 2x2 + 4y2
0
50
100
150
22 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Upper Level Set Example
140140
140140
130130
130130
120120
120120
110110
110110
100 100
100
100100
9090
90
9090
90
8080
80
8080
80
7070
70
7070
70
6060
60
6060
6050
50
50
50
50 50
50
40
40
40
40
40
40
30
3030
30
30
20
20
20
20
10
10
10
−4 −2 0 2 4
−4
−2
0
2
4
x
y
Plot of 2x2 + 4y2
23 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Example
0
2
40
2
4
0
100
200
x y
zPlot of 54x+−9x2 + 78y − 13y2
0
50
100
150
24 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Example
180
180
180
180
160160
160
160160
160
140140
140
140 140
120
120
120
120120
100
100100
100
80
80
80
60604020
0 1 2 3 4 50
1
2
3
4
5
x
y
Plot of 54x+−9x2 + 78y − 13y2
25 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Convexity, Concavity, and Optima
Theorem
Suppose that S is convex and that f(x) is convex on S for the problemminx∈S f(x), then
• If x∗ is locally minimal, then x∗ is globally minimal
• The set X∗ of global optimal solutions is convex
• If f is strictly convex, then x∗ is unique
26 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Examples
Problem 1
Minimize −x2lnx1 + x1
9 + x22Subject to: 1.0 ≤ x1 ≤ 5.0
0.6 ≤ x2 ≤ 3.6
27 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
What does the function look like?
−20
2 −2
0
2−1
0
1
xy
zz= 7xy
exp(x2+y2)
−1
−0.5
0
0.5
1
28 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Problem 2
Minimize∑3
i=1−ilnxiSubject to:
∑3i=1 xi = 6
xi ≤ 3.5 i = 1, 2, 3
xi ≥ 1.5 i = 1, 2, 3
29 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Class exercises
• Show that f(x) = ||x|| =√∑
i x2i is convex
• Prove that any level set of a convex function is a convex set
30 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Other Types of Convexity
• The idea of pseudoconvexity of a function is to extend the class of functions forwhich stationarity is a sufficient condition for global optimality. If f is defined onan open set X and is differentiable we define the concept of pseudoconvexity.
• A differentiable function f is pseudoconvex if
∇f(x) · (x′ − x) ≥ 0⇒ f(x′) ≥ f(x) ∀x, x′ ∈ X
• or alternatively ..
f(x′) < f(x)⇒ ∇f(x)(x′ − x) < 0 ∀x, x′ ∈ X
• A function f is pseudoconcave iff −f is pseudoconvex
• Note that if f is convex and differentiable, and X is open, then f is alsopseudoconvex
31 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Other Types of Convexity
• A function is quasiconvex if all lower level sets are convex
• That is, the following sets are convex
S′ = {x : f(x) ≤ β}
• A function is quasiconcave if all upper level sets are convex
• That is, the following sets are convex
S′ = {x : f(x) ≥ β}
• Note that if f is convex and differentiable, and X is open, then f is alsoquasiconvex
• Convexity properties
Convex ⇒ pseudoconvex ⇒ quasiconvex
32 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Exercises
Show the following
• f(x) = x+ x3 is pseudoconvex but not convex
• f(x) = x3 is quasiconvex but not pseudoconvex
33 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Graphically
1.2
1.2
1
1
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2 0.2
0.2
0.2
0.2
0.2
0
00
0
−0.2
−0.2
−0.2
−0.2
−0.2
−0.2
−0.4
−0.4
−0.4
−0.4
−0.6
−0.6
−0.6
−0.6
−0.8
−0.8−1
−1
−1.2
−1.2
−2 −1 0 1 2
−2
−1
0
1
2
x
y
z= 7xyexp(x2+y2)
34 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Exercises
Convexity Questions
• Can a function be both convex and concave?
• Is a convex function of a convex function convex?
• Is a convex combination of convex functions convex?
• Is the intersection of convex sets convex?
35 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Unconstrained problem
min f(x) s.t. x ∈ Rn
• Necessary optimality condition for xo to be a local minimum
∇f(xo) = 0 and H(xo) is positive semidefinite
• Sufficient optimality condition for xo to be a local minimum
∇f(xo) = 0 and H(xo) is positive definite
• Necessary and sufficient
• Suppose f is pseudoconvex
• x∗ is a global minimum iff ∇f(x∗) = 0
36 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Unconstrained example
min f(x) = (x2 − 1)3
• f ′(x) = 6x(x2 − 1)2 = 0 for x = 0,±1• H(x) = 24x2(x2 − 1) + 6(x2 − 1)2
• H(0) = 6 and H(±1) = 0
• Therefore x = 0 is a local minimum (actually the global minimum)
• x = ±1 are saddle points
37 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
What does the function look like?
12
34
5 12
34
5
0
20
x1 x2
f(x)
Plot of − x2ln(x1) +x19 + x22
0
5
10
15
20
25
38 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Class Exercise
Problem
Suppose A is an m ∗ n matrix, b is a given m vector, find
min ||Ax− b||2
39 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017
Richard M. Lusby
DTU Management Engineering, Technical University of Denmark
Building 424, Room 208 [email protected]
2800 Kgs. Lyngby, Denmark phone +45 4525 3084
http://www.man.dtu.dk
40 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017