Upload
buikhanh
View
230
Download
0
Embed Size (px)
Citation preview
Duality
I Lagrange dual problem
I weak and strong duality
I geometric interpretation
I optimality conditions
I perturbation and sensitivity analysis
I examples
I generalized inequalities
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–1
Lagrangian
Standard form problem (not necessarily convex)
minimize f0(x)subject to f
i
(x) 0, i = 1, . . . ,mhi
(x) = 0, i = 1, . . . , p
variable x 2 Rn, domain D 6= ;, optimal value p?
Lagrangian: L : Rn ⇥Rm ⇥Rp ! R, with dom L = D ⇥Rm ⇥Rp,
L(x , �, ⌫) = f0(x) +mX
i=1
�i
fi
(x) +pX
i=1
⌫i
hi
(x)
I weighted sum of objective and constraint functions
I �i
is Lagrange multiplier associated with fi
(x) 0
I ⌫i
is Lagrange multiplier associated with hi
(x) = 0
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–2
Lagrange dual function
Lagrange dual function: g : Rm ⇥ Rp ! R,
g(�, ⌫) = infx2D
L(x , �, ⌫)
= infx2D
f0(x) +
mX
i=1
�i
fi
(x) +pX
i=1
⌫i
hi
(x)
!
g is concave, can be �1 for some �, ⌫
Lower bound property: if � ⌫ 0, then g(�, ⌫) p?
Proof: if x is feasible and � ⌫ 0, then
f0(x) � L(x , �, ⌫) � infx2D
L(x , �, ⌫) = g(�, ⌫).
Minimizing over all feasible x gives p? � g(�, ⌫)
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–3
Example: Least-norm solution of linear equations
minimize xT xsubject to Ax = b
dual function
I Lagrangian is L(x , ⌫) = xT x + ⌫T (Ax � b)
I to minimize L over x , set gradient equal to zero:
rx
L(x , ⌫) = 2x + AT⌫ = 0 =) x(⌫) = �(1/2)AT⌫
I plug it into L to obtain g :
g(⌫) = L((�1/2)AT⌫, ⌫) = �1
4⌫TAAT⌫ � bT⌫
— a concave function of ⌫
lower bound property: p? � �(1/4)⌫TAAT⌫ � bT⌫ for all ⌫IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–4
Example: Standard form LP
minimize cT xsubject to Ax = b, x ⌫ 0
dual functionI Lagrangian is
L(x , �, ⌫) = cT x + ⌫T (Ax � b) � �T x
= �bT⌫ + (c + AT⌫ � �)T x
I L is linear in x , hence
g(�, ⌫) = infx
L(x , �, ⌫) =
⇢ �bT⌫ AT⌫ � � + c = 0�1 otherwise
g is linear on a�ne domain {(�, ⌫) | AT⌫ � � + c = 0}, henceconcave
lower bound property: p? � �bT⌫ if AT⌫ + c ⌫ 0IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–5
Example: Equality constrained norm minimization
minimize kxksubject to Ax = b
dual function
g(⌫) = infx
(kxk � ⌫TAx + bT⌫) =
⇢bT⌫ kAT⌫k⇤ 1�1 otherwise
,
where kvk⇤ = supkuk1 uT v is the dual norm of k · k
Proof: consider infx
(kxk � yT x), where y = AT⌫.
I If kyk⇤ 1, then kxk � yT x � 0 for all x , with equality ifx = 0, so inf
x
(kxk � yT x) = 0.I If kyk⇤ > 1, choose x = tu where kuk 1, uT y = kyk⇤ > 1:
kxk � yT x = t(kuk � kyk⇤) ! �1 as t ! 1,
so infx
(kxk � yT x) = �1.
lower bound property: p? � bT⌫ if kAT⌫k⇤ 1IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–6
Two-way partitioning
minimize xTWxsubject to x2
i
= 1, i = 1, . . . , n
I a nonconvex problem; feasible set contains 2n discrete points
I interpretation: partition {1, . . . , n} in two sets; Wij
is cost ofassigning i , j to the same set; �W
ij
is cost of assigning todi↵erent sets
dual function
g(⌫) = infx
(xTWx +X
i
⌫i
(x2i
� 1)) = infx
xT (W + diag(⌫))x � 1T⌫
=
⇢ �1T⌫ W + diag(⌫) ⌫ 0�1 otherwise
lower bound property: p? � �1T⌫ if W + diag(⌫) ⌫ 0example: ⌫ = ��
min
(W )1 gives bound p? � n�min
(W )IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–7
The conjugate function
the conjugate of a function f is
f ⇤(y) = supx2dom f
(yT x � f (x))
The conjugate function
the conjugate of a function f is
f∗(y) = supx∈dom f
(yTx − f(x))
PSfrag replacements
f(x)
(0,−f∗(y))
xy
x
• f∗ is convex (even if f is not)
• will be useful in chapter 5
Convex functions 3–21
I f ⇤ is convex (even if f is not)
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–8
Examples
I strictly convex quadratic f (x) = (1/2)xTQx with Q 2 Sn
++
f ⇤(y) = supx
(yT x � (1/2)xTQx)
=1
2yTQ�1y
I negative logarithm f (x) = � log x
f ⇤(y) = supx>0
(xy + log x)
=
⇢ �1 � log(�y) y < 01 otherwise
I log-determinant: f (X ) = log detX�1 on Sn
++.f ⇤(Y ) = log det(�Y )�1 � n with dom f ⇤ = �Sn
++.
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–9
Lagrange dual function and conjugate functionRecall: conjugate function f ⇤(y) = sup
x2dom f
(yT x � f (x))
minimize f0(x)subject to Ax � b, Cx = d
dual function
g(�, ⌫) = infx2dom f0
⇣f0(x) + (AT� + CT⌫)T x � bT� � dT⌫
⌘
= �f ⇤0 (�AT� � CT⌫) � bT� � dT⌫
example: minimum-volume covering ellipsoid
minimize f0(X ) = log detX�1 subject to aTi
Xai
1, i = 1, . . . ,m
I f ⇤0 (Y ) = log det(�Y )�1 � n with dom f ⇤0 = �Sn
++
I aTi
Xai
1 , tr((ai
aTi
)X ) 1
g(�) =
(log det(
Pm
i=1 �i
ai
aTi
) � 1T� + n,P
i
�i
ai
aTi
� 0,
�1 otherwise.
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–10
The dual problemLagrange dual problem
maximize g(�, ⌫)subject to � ⌫ 0
I finds best lower bound on p?, obtained from Lagrange dualfunction
I a convex optimization problem; optimal value denoted d?
I �, ⌫ are dual feasible if � ⌫ 0, (�, ⌫) 2 dom g
I often simplified by making implicit constraint (�, ⌫) 2 dom gexplicit
example: standard form LP and its dual
minimize cT xsubject to Ax = b
x ⌫ 0
maximize �bT⌫subject to AT⌫ + c ⌫ 0
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–11
Weak and strong dualityweak duality: d? p?
I always holds (for convex and nonconvex problems), even if p?
and/or d? are infinite.
I can be used to find nontrivial lower bounds for di�cultproblemsfor example, solving the SDP
maximize �1T⌫subject to W + diag(⌫) ⌫ 0
gives a lower bound for the two-way partitioning problem
strong duality: d? = p?
I does not hold in general
I (usually) holds for convex problems
I conditions that guarantee strong duality in convex problemsare called constraint qualifications
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–12
Slater’s constraint qualification
strong duality holds for a convex problem
minimize f0(x)subject to f
i
(x) 0, i = 1, . . . ,mAx = b
if it is strictly feasible, i.e.,
9x 2 intD : fi
(x) < 0, i = 1, . . . ,m, Ax = b
I also guarantees that the dual optimum is attained (ifp? > �1)
I can be sharpened: e.g., can replace intD with rel intD(interior relative to a�ne hull); linear inequalities do not needto hold with strict inequality, . . .
I there exist many other types of constraint qualifications
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–13
Inequality form LP
primal problemminimize cT xsubject to Ax � b
dual function
g(�) = infx
⇣(c + AT�)T x � bT�
⌘=
⇢ �bT� AT� + c = 0�1 otherwise
dual problem
maximize �bT�subject to AT� + c = 0, � ⌫ 0
I Convex problem with linear constraints, so p? = d?
I ... except when both primal and dual are infeasible
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–14
Quadratic program
primal problem (assume P 2 Sn
++)
minimize xTPxsubject to Ax � b
dual function
g(�) = infx
⇣xTPx + �T (Ax � b)
⌘= �1
4�TAP�1AT� � bT�
dual problem
maximize �(1/4)�TAP�1AT� � bT�subject to � ⌫ 0
I Convex problem with linear constraints, so p? = d?
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–15
A nonconvex problem with strong duality
minimize xTAx + 2bT xsubject to xT x 1
nonconvex if A 6⌫ 0dual function: g(�) = inf
x
(xT (A + �I )x + 2bT x � �)
I unbounded below if A + �I 6⌫ 0 or if A + �I ⌫ 0 andb 62 R(A + �I )
I minimized by x = �(A + �I )†b otherwise:g(�) = �bT (A + �I )†b � �
dual problem and equivalent SDP:
maximize �bT (A + �I )†b � �subject to A + �I ⌫ 0
b 2 R(A + �I )
maximize �t � �
subject to
A + �I bbT t
�⌫ 0
strong duality although primal problem is not convex (not easy toshow)
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–16
Geometric interpretationfor simplicity, consider problem with one constraint f1(x) 0interpretation of dual function:
g(�) = inf(u,t)2G
(t + �u), where G = {(f1(x), f0(x)) | x 2 D}
Geometric interpretation
for simplicity, consider problem with one constraint f1(x) ≤ 0
interpretation of dual function:
g(λ) = inf(u,t)∈G
(t + λu), where G = {(f1(x), f0(x)) | x ∈ D}
PSfrag replacementsG
f1(x) p⋆
g(λ)λu + t = g(λ)
t
u
PSfrag replacements
G
p⋆
d⋆
t
u
• λu + t = g(λ) is (non-vertical) supporting hyperplane to G• hyperplane intersects t-axis at t = g(λ)
Duality 5–15
I G is the set of values taken on by constraint and objectivefunctions
I For � ⌫ 0, �u + t = g(�) is (non-vertical) supportinghyperplane to G
I hyperplane intersects t-axis at t = g(�)IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–17
Epigraph variationSame interpretation if G is replaced with
A = {(u, t) | f1(x) u, f0(x) t for some x 2 D}epigraph variation: same interpretation if G is replaced with
A = {(u, t) | f1(x) u, f0(x) t for some x 2 D}
PSfrag replacementsA
f1(x)p⋆
g(λ)
λu + t = g(λ)
t
u
strong duality
• holds if there is a non-vertical supporting hyperplane to A at (0, p?)
• for convex problem, A is convex, hence has supp. hyperplanes at (0, p?)
• Slater’s condition: if there exist (u, t) 2 A with u < 0, then supportinghyperplanes at (0, p?) must be non-vertical
Duality 5–16
I A = {(u, v , t) | f (x) � u, h(x) = v , f0(x) t for some x 2 D}I p? = inf{t | (0, 0, t) 2 A}I g(�, ⌫) = inf{(�, ⌫, 1)T (u, v , t) | (u, v , t) 2 A}
I if the inf is finite, (�, ⌫, 1)T (u, v , t) � g(�, ⌫) is a non-verticalsupporting hyperplane to A (weak duality).
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–18
Strong duality
From the geometric interpretation with one inequality:
I Strong duality holds if there is a non-vertical supportinghyperplane to A at (0, p?)
I For convex problem, A is convex, hence has suppoprtinghyperplane at (0, p?)
I Slater’s condition: if there exist (u, t) 2 A with u < 0, thensupporting hyperplanes at (0, p?) must be non-vertical
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–19
Proof of strong duality under convexity and Slater’s CQTo simplify, assume:
I A has full row rank (wolog for feasible problems)
I 9x 2 intD: Ax = b, fi
(x) < 0, i = 1, . . . ,m (Slater point)
I p? is finite
Outline of the proof:
I A is convex; B = {(0, 0, s) | s < p?} is convex, A \ B = ;.
I There exists a hyperplane separating A and BI Interpretation: 9� � 0, µ � 0, ⌫:
�T f (x) + ⌫T (Ax � b) + µf0(x) � µp? 8x 2 D
I If µ > 0, g(�/µ, ⌫/µ) = p? — strong duality! (and we foundthe solution of the dual)
I If µ = 0, use the Slater point and the first assumption toderive the contradiction.
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–20
Complementary slacknessAssume strong duality holds, x? is primal optimal, (�?, ⌫?) is dualoptimal (not assuming primal problem is convex!)
f0(x?) = g(�?, ⌫?) = inf
x
f0(x) +
mX
i=1
�?i
fi
(x) +pX
i=1
⌫?i
hi
(x)
!
f0(x?) +
mX
i=1
�?i
fi
(x?) +pX
i=1
⌫?i
hi
(x?)
f0(x?)
hence, the two inequalities hold with equalityI x? minimizes L(x , �?, ⌫?)I �?
i
fi
(x?) = 0 for i = 1, . . . ,m (known as complementaryslackness):
�?i
> 0 =) fi
(x?) = 0, fi
(x?) < 0 =) �?i
= 0
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–21
Karush-Kuhn-Tucker (KKT) conditionsFor a problem with di↵erentiable f
i
’s and hi
’s:Assume strong duality holds, x is primal optimal, (�, ⌫) is dualoptimal (still no convexity assumed in the primal). Then:
1. primal constraints: fi
(x) 0, i = 1, . . . ,m, hi
(x) = 0,i = 1, . . . , p
2. dual constraints: � ⌫ 0
3. complementary slackness: �i
fi
(x) = 0, i = 1, . . . ,m
4. gradient of Lagrangian with respect to x vanishes:
rf0(x) +mX
i=1
�i
rfi
(x) +pX
i=1
⌫i
rhi
(x) = 0
The above four conditions are called KKT conditions.
If strong duality holds and x , �, ⌫ are optimal, then they mustsatisfy the KKT conditions
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–22
Implication of KKT conditions for convex problems
if x , �, ⌫ satisfy KKT for a convex problem, then they are optimal:
I from complementary slackness: f0(x) = L(x , �, ⌫)
I from 4th condition (and convexity): g(�, ⌫) = L(x , �, ⌫)
hence, f0(x) = g(�, ⌫)
If Slater’s condition is satisfied:x is optimal if and only if there exist �, ⌫ that satisfy KKTconditions
I recall that Slater implies strong duality, and dual optimum isattained
I generalizes optimality condition rf0(x) = 0 for unconstrainedproblem
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–23
Example(assume ↵
i
> 0)
minimize �Pn
i=1 log(xi
+ ↵i
)subject to x ⌫ 0, 1T x = 1
x is optimal i↵ x ⌫ 0, 1T x = 1, and there exist � 2 Rn, ⌫ 2 Rsuch that
� ⌫ 0, �i
xi
= 0,1
xi
+ ↵i
+ �i
= ⌫
I if ⌫ < 1/↵i
: �i
= 0 and xi
= 1/⌫ � ↵i
I if ⌫ � 1/↵i
: �i
= ⌫ � 1/↵i
and xi
= 0I determine ⌫ from 1T x =
Pn
i=1 max{0, 1/⌫ � ↵i
} = 1interpretation: water-filling
I n patches; level of patch i is at height ↵i
I flood area with unit amount of water
I resulting level is 1/⌫?
example: water-filling (assume αi > 0)
minimize −!n
i=1 log(xi + αi)subject to x ≽ 0, 1Tx = 1
x is optimal iff x ≽ 0, 1Tx = 1, and there exist λ ∈ Rn, ν ∈ R such that
λ ≽ 0, λixi = 0,1
xi + αi+ λi = ν
• if ν < 1/αi: λi = 0 and xi = 1/ν − αi
• if ν ≥ 1/αi: λi = ν − 1/αi and xi = 0
• determine ν from 1Tx =!n
i=1 max{0, 1/ν − αi} = 1
interpretation
• n patches; level of patch i is at height αi
• flood area with unit amount of water
• resulting level is 1/ν⋆
PSfrag replacements
i
1/ν⋆
xi
αi
Duality 5–20IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–24
Feasibility problems and alternative systemsfeasibility problem A (variables x 2 Rn)
fi
(x) < 0, i = 1, . . . ,m, hi
(x) = 0, i = 1, . . . , p
feasibility problem B (variables � 2 Rm, ⌫ 2 Rp)
� ⌫ 0, � 6= 0, g(�, ⌫) � 0
where g(�, ⌫) = infx
�Pm
i=1 �i
fi
(x) +P
p
i=1 ⌫i
hi
(x)�
I feasibility problem B is convex even if problem A is notI A and B are always weak alternatives: at most one is
feasibleproof: assume x satisfies A, �, ⌫ satisfy B
0 g(�, ⌫) mX
i=1
�i
fi
(x) +pX
i=1
⌫i
hi
(x) < 0
I A and B are strong alternatives if exactly one of the two isfeasible
I can prove infeasibility of A by producing solution of B andvice-versa
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–25
Strong alternatives: convex case
(A) fi
(x) < 0, i = 1, . . . ,m, hi
(x) = 0, i = 1, . . . , p(B) � ⌫ 0, � 6= 0, g(�, ⌫) � 0
I Suppose A is convex: fi
’s are convex, h(x) = Ax � b 2 Rp
I Suppose further 9x 2 rel intD : Ax = bI Theorem: A and B are strong alternatives
I Weak alternatives — already established.I Consider optimization problem (p? < 0 i↵ A has a solution)
minimizex,s s
subject to f (x) � 1s � 0Ax = b
I Dual (g(�, ⌫) as in B): attains d? = p?, due to Slater’s CQ
maximize�,⌫ g(�, ⌫)subject to � ⌫ 0, 1T� = 1
I If A is infeasible, d? � 0 and 9(�, ⌫) that satisfy BIOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–26
Example: homogeneous linear inequalities
I (A) Fx < 0, Cx 0, Ax = 0I g(�, µ, ⌫) = inf
x
(FT� + CTµ + AT⌫)T x =(0 FT� + CTµ + AT⌫ = 0
�1 otherwise.
I (B) (�, µ) ⌫ 0, � 6= 0, FT� + CTµ + AT⌫ = 0
I Farkas’ lemma: Ax 0, cT x < 0 and AT y + c = 0, y ⌫ 0are strong alternatives
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–27
Example: first-order optimality conditions
minimize f0(x) subj. to fi
(x) 0, i = 1, . . . ,m, hi
(x) = 0, i = 1, . . . , p
I Assume di↵erentiable functions, do not assume convexityI Fritz-John necessary optimality conditions: if x is a local
minimizer, then 9(�0, �, ⌫) 6= 0 : (�0, �) ⌫ 0, �i
fi
(x) = 0 8i :
�0rf0(x) +mX
i=1
�i
rfi
(x) +pX
i=1
⌫i
rhi
(x) = 0
I If rhi
(x), i = 1, . . . , p are linearly dependent, set (�0, �) = 0I If rh
i
(x), i = 1, . . . , p are linearly independent,I let I ⌘ {i > 0 : f
i
(x) = 0}I then 6 9d : rf
i
(x)Td < 0, i 2 I [ {0}; rh
i
(x)Td = 0, 1 i p
I True i↵ 9(�0,�, ⌫) : (�0,�) ⌫ 0, (�0,�) 6= 0,
�0rf0(x) +mX
i=1
�i
rf
i
(x) +
pX
i=1
⌫i
rh
i
(x) = 0
I If also rhi
(x), i = 1, . . . , p, rfi
(x), i 2 I are lin. ind., then�0 > 0
I Result: KKT necessary optimality conditions (with a lin. ind. CQ)IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–28
Duality and problem reformulations
I equivalent formulations of a problem can lead to very di↵erentduals
I reformulating the primal problem can be useful when the dualis di�cult to derive, or uninteresting
Common reformulations
I introduce new variables and equality constraints
I make explicit constraints implicit or vice-versa
I transform objective or constraint functionse.g., replace f0(x) by �(f0(x)) with � convex, increasing
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–29
Introducing new variables and equality constraints
minimize f0(Ax + b)
I dual function is constant:g = inf
x
L(x) = infx
f0(Ax + b) = p?
I we have strong duality, but dual is quite useless
Reformulated problem and its dual
minimize f0(y)subject to Ax + b � y = 0
maximize bT⌫ � f ⇤0 (⌫)subject to AT⌫ = 0
Dual function follows from
g(⌫) = infx ,y
(f0(y) � ⌫T y + ⌫TAx + bT⌫)
=
⇢ �f ⇤0 (⌫) + bT⌫ AT⌫ = 0�1 otherwise
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–30
Norm approximation problem:minimize kAx � bk
minimize kyksubject to y = Ax � b
can look up conjugate of k · k, or derive dual directly
g(⌫) = infx ,y
(kyk + ⌫T y � ⌫TAx + bT⌫)
=
⇢bT⌫ + inf
y
(kyk + ⌫T y) AT⌫ = 0�1 otherwise
=
⇢bT⌫ AT⌫ = 0, k⌫k⇤ 1�1 otherwise
(see page 5-4)Dual of norm approximation problem
maximize bT⌫subject to AT⌫ = 0, k⌫k⇤ 1
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–31
Implicit constraintsLP with box constraints: primal and dual problem
minimize cT xsubject to Ax = b
�1 � x � 1
maximize �bT⌫ � 1T�1 � 1T�2
subject to c + AT⌫ + �1 � �2 = 0�1 ⌫ 0, �2 ⌫ 0
Reformulation with box constraints made implicit
minimize f0(x) =
⇢cT x �1 � x � 11 otherwise
subject to Ax = b
Dual function
g(⌫) = inf�1�x�1
(cT x + ⌫T (Ax � b))
= �bT⌫ � kAT⌫ + ck1Dual problem: maximize �bT⌫ � kAT⌫ + ck1
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–32
Problems with generalized inequalities
minimize f0(x)subject to f
i
(x) �K
i
0, i = 1, . . . ,mhi
(x) = 0, i = 1, . . . , p
�K
i
is generalized inequality on Rk
i
Definitions are parallel to scalar case:
I Lagrange multiplier for fi
(x) �K
i
0 is vector �i
2 Rk
i
I Lagrangian L : Rn ⇥ Rk1 ⇥ · · · ⇥ Rk
m ⇥ Rp ! R, is defined as
L(x , �1, · · · , �m
, ⌫) = f0(x) +mX
i=1
�T
i
fi
(x) +pX
i=1
⌫i
hi
(x)
I dual function g : Rk1 ⇥ · · · ⇥ Rk
m ⇥ Rp ! R, is defined as
g(�1, . . . , �m
, ⌫) = infx2D
L(x , �1, · · · , �m
, ⌫)
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–33
Lower bound property: if �i
⌫K
⇤i
0, then g(�1, . . . , �m
, ⌫) p?
Proof: if x is feasible and � ⌫K
⇤i
0, then
f0(x) � f0(x) +mX
i=1
�T
i
fi
(x) +pX
i=1
⌫i
hi
(x)
� infx2D
L(x , �1, . . . , �m
, ⌫)
= g(�1, . . . , �m
, ⌫)
minimizing over all feasible x gives p? � g(�1, . . . , �m
, ⌫)Dual problem
maximize g(�1, . . . , �m
, ⌫)subject to �
i
⌫K
⇤i
0, i = 1, . . . ,m
I weak duality: p? � d? alwaysI strong duality: p? = d? for convex problem with constraint
qualification(for example, Slater’s: primal problem is strictly feasible)
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–34
Semidefinite programprimal SDP (F
i
,G 2 Sk)
minimize cT xsubject to x1F1 + · · · + x
n
Fn
� G
I Lagrange multiplier is matrix Z 2 Sk
I Lagrangian L(x ,Z ) = cT x + tr (Z (x1F1 + · · · + xn
Fn
� G ))I dual function
g(Z ) = infx
L(x ,Z ) =
(� tr(GZ ) tr(F
i
Z ) + ci
= 0, i = 1, . . . , n,
�1 otherwise.
dual SDP
maximize � tr(GZ )subject to Z ⌫ 0, tr(F
i
Z ) + ci
= 0, i = 1, . . . , n
p? = d? if primal SDP is strictly feasible (9x withx1F1 + · · · + x
n
Fn
� G )IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–35
Alternative systems with generalized inequalitiesI Suppose f
i
(x)’s are Ki
-convex (Ki
’s are proper cones) andhi
(x) are a�ne:
(A) fi
(x) �K
i
0, i = 1, . . . ,m, Ax = b
I g(�, ⌫) = infx2D(
Pm
i=1 �T
i
fi
(x) + ⌫T (Ax � b))
(B) �i
⌫K
?i
0, i = 1, . . . ,m, � 6= 0, g(�, ⌫) � 0
I A and B are weak alternatives (easy to show)I If, in addition, 9x 2 rel intD : Ax = b, they are strong
alternativesI To prove, consider optimization problem (e
i
�K
i
0 8i)minimize
x ,s ssubject to f
i
(x) �K
i
sei
, i = 1, . . . ,mAx = b
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–36
Example: feasibility of an LMI
(A) F (x) = x1F1 + · · · + xn
Fn
+ G � 0
(B) Z ⌫ 0, Z 6= 0, tr(GZ ) � 0, tr(Fi
Z ) = 0, i = 1, . . . , n
I The above systems are strong alternativesI Similar analysis can be done for A with non-strict inequality:
I Need conditions on matrices Fi
, e.g.,
nX
i=1
vi
Fi
⌫ 0 =)nX
i=1
vi
Fi
= 0
I If this condition holds, the following are strong alternatives:
(A) F (x) = x1F1 + · · · + xn
Fn
+ G � 0
(B) Z ⌫ 0, tr(GZ ) > 0, tr(Fi
Z ) = 0, i = 1, . . . , n
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–37
Perturbation and sensitivity analysis(Unperturbed) optimization problem and its dual
minimize f0(x)subject to f
i
(x) 0, i = 1, . . . ,mhi
(x) = 0, i = 1, . . . , p
maximize g(�, ⌫)subject to � ⌫ 0
Perturbed problem and its dual
min. f0(x)s.t. f
i
(x) ui
, i = 1, . . . ,mhi
(x) = vi
, i = 1, . . . , p
max. g(�, ⌫) � uT� � vT⌫s.t. � ⌫ 0
I x is primal variable; u, v are parametersI p?(u, v) is optimal value as a function of (u, v)
(p?(0, 0) = p?)I we are interested in information about p?(u, v) that we can
obtain from the solution of the unperturbed problem and itsdual
IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–38
Global sensitivity resultAssume strong duality holds for unperturbed problem, and that �?,⌫? are dual optimal for unperturbed problem.Apply weak duality to perturbed problem:
p?(u, v) � g(�?, ⌫?) � uT�? � vT⌫?
= p?(0, 0) � uT�? � vT⌫?
sensitivity interpretationI if �?
i
large: p? increases greatly if we tighten constraint i(u
i
< 0)I if �?
i
small: p? does not decrease much if we loosen constrainti (u
i
> 0)I if ⌫?
i
large and positive: p? increases greatly if we take vi
< 0;if ⌫?
i
large and negative: p? increases greatly if we take vi
> 0I if ⌫?
i
small and positive: p? does not decrease much if we takevi
> 0;if ⌫?
i
small and negative: p? does not decrease much if wetake v
i
< 0IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–39
Local sensitivity:If (in addition) p?(u, v) is di↵erentiable at (0, 0), then
�?i
= �@p?(0, 0)
@ui
, ⌫?i
= �@p?(0, 0)
@vi
Proof (for �?i
): from global sensitivity result,
@p?(0, 0)
@ui
= limt&0
p?(tei
, 0) � p?(0, 0)
t� ��?
i
@p?(0, 0)
@ui
= limt%0
p?(tei
, 0) � p?(0, 0)
t ��?
i
hence, equality.
p?(u) for a problem with one(inequality) constraint:
local sensitivity: if (in addition) p⋆(u, v) is differentiable at (0, 0), then
λ⋆i = −∂p⋆(0, 0)
∂ui, ν⋆
i = −∂p⋆(0, 0)
∂vi
proof (for λ⋆i ): from global sensitivity result,
∂p⋆(0, 0)
∂ui= lim
t↘0
p⋆(tei, 0) − p⋆(0, 0)
t≥ −λ⋆
i
∂p⋆(0, 0)
∂ui= lim
t↗0
p⋆(tei, 0) − p⋆(0, 0)
t≤ −λ⋆
i
hence, equality
p⋆(u) for a problem with one (inequality)constraint:
PSfrag replacements
up⋆(u)
p⋆(0) − λ⋆u
u = 0
Duality 5–23IOE 611: Nonlinear Programming, Fall 2017 6. Duality Page 6–40