Nonsmooth, Nonconvex Optimization€¦ · Convex and Nonsmooth Optimization Class, Spring 2018,...

1 / 71

Nonsmooth, Nonconvex OptimizationAlgorithms and Examples

Michael L. OvertonCourant Institute of Mathematical Sciences

New York University

Convex and Nonsmooth Optimization Class, Spring 2018, Final Lecture

Based on my research work with Jim Burke (Washington), Adrian Lewis (Cornell)

and others

Introduction

IntroductionNonsmooth,NonconvexOptimization

A Simple NonconvexExample

Failure of GradientDescent inNonsmooth CaseArmijo-Wolfe LineSearchFailure of GradientMethod: SimpleConvex Example

Illustration of Failureand SuccessMethods Suitable forNonsmoothFunctions

Gradient Sampling

Quasi-NewtonMethods

A DifficultNonconvex Problemfrom Nesterov

Limited MemoryMethods

Concluding Remarks

2 / 71

Nonsmooth, Nonconvex Optimization

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

Problem: find x that locally minimizes f , where f : Rn → R is

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

■ Continuous

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

■ Continuous■ Not differentiable everywhere, in particular often not

differentiable at local minimizers

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

differentiable at local minimizers■ Not convex

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

differentiable at local minimizers■ Not convex■ Usually, but not always, locally Lipschitz: for all x there

exists Lx s.t. |f(y)− f(z)| ≤ Lx‖y − z‖ for all y, z near x

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

exists Lx s.t. |f(y)− f(z)| ≤ Lx‖y − z‖ for all y, z near x■ Otherwise, no structure assumed

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

Lots of interesting applications

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

Any locally Lipschitz function is differentiable almost everywhereon its domain. So, whp, can evaluate gradient at any given point.

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

3 / 71

Any locally Lipschitz function is differentiable almost everywhereon its domain. So, whp, can evaluate gradient at any given point.

What happens if we simply use gradient descent (steepestdescent) with a standard line search?

A Simple Nonconvex Example

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

4 / 71

f(x)=10*|x2 − x

12| + (1−x

steepest descent iterates−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

A Simple Nonconvex Example

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

4 / 71

f(x)=10*|x2 − x

12| + (1−x

steepest descent iterates−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

On this example, iterates invariably converge to a nonstationary point

Failure of Gradient Descent in Nonsmooth Case

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

5 / 71

Known for decades that gradient descent may converge tononstationary points when f is nonsmooth, even if it is convex.

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

5 / 71

■ V.F. Dem’janov and V.N. Malozemov, 1970■ P. Wolfe, 1975■ J.-B. Hiriart-Urruty and C. Lemarechal, 1993

But these are all examples cooked up to defeat exact linesearches from a specific starting point.

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

5 / 71

■ V.F. Dem’janov and V.N. Malozemov, 1970■ P. Wolfe, 1975■ J.-B. Hiriart-Urruty and C. Lemarechal, 1993

But these are all examples cooked up to defeat exact linesearches from a specific starting point.

Failure can be avoided by using sufficiently short steplengths(N.Z. Shor, 1970s), but this is slow.

Armijo-Wolfe Line Search

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

6 / 71

Given x with f differentiable at x and d with ∇f(x)T d < 0, andparameters 0 < c1 < c2 < 1, find steplength t so that

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

6 / 71

■ sufficient decrease in function value:f(x+ td) < f(x) + c1t∇f(x)Td (L. Armijo, 1966)

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

6 / 71

■ sufficient increase in directional derivative: f is differentiable atx+ td and ∇f(x+ td)T d > c2∇f(x)Td (P. Wolfe, 1969)

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

6 / 71

Assuming inft f(x+ td) is bounded below,

■ the Armijo condition holds for sufficiently small t as long as f iscontinuous

■ the Wolfe condition holds for sufficiently large t as long as f isdifferentiable

■ the intervals where each holds overlap

so combining the two conditions leads to a convenient, convergentbracketing line search (M.J.D. Powell, 1976)

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

6 / 71

Extends to locally Lipschitz case (A.S. Lewis and M.L.O., 2013)

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

6 / 71

Extends to locally Lipschitz case (A.S. Lewis and M.L.O., 2013)

Searching for “Armijo-Wolfe” on the web, we foundMelissa Armijo-Wolfe’s LinkedIn page!

Failure of Gradient Method: Simple Convex Example

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

7 / 71

Let f(x) = a|x1|+ x2, with a ≥ 1. Although f is unboundedbelow, it is bounded below along any direction d = −∇f(x).

-5-10-10

Failure of Gradient Method: Simple Convex Example

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

7 / 71

Let f(x) = a|x1|+ x2, with a ≥ 1. Although f is unboundedbelow, it is bounded below along any direction d = −∇f(x).

-5-10-10

Theorem. Let x(0) satisfy x(0)1 6= 0 and define x(k) ∈ R

x(k+1) = x(k) + tkd(k) where d(k) = −∇f(x(k))

and tk is any steplength satisfying the Armijo and Wolfeconditions with Armijo parameter c1. If

c1(a2 + 1) > 1

then x(k) converges to x with x1 = 0, even though f isunbounded below.

Azam Asl and M.L.O., 2017

Illustration of Failure and Success

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

8 / 71

u-3 -2 -1 0 1 2 3

6f(u,v) = 5|u|+v. x_0 = (-2.264; 5), c_1=0.1, τ =0.064

u-3 -2 -1 0 1 2 3

6f(u,v) = 2|u|+v. x_0 = (-2.264; 5), c_1=0.1, τ =-0.125

a = 5, c1 = 0.1 a = 2, c1 = 0.1

x(k) → x f(x(k)) ↓ −∞

Methods Suitable for Nonsmooth Functions

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

9 / 71

Exploit the gradient information obtained at several points, notjust at one point:

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

9 / 71

■ Bundle methods (C. Lemarechal, K.C. Kiwiel, 1980s –)extensive practical use and theoretical analysis, butcomplicated in nonconvex case

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

9 / 71

■ Gradient sampling: an easily stated method with niceconvergence theory (J.V. Burke, A.S. Lewis, M.L.O., 2005;K.C. Kiwiel, 2007), but computationally intensive

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

9 / 71

■ Gradient sampling: an easily stated method with niceconvergence theory (J.V. Burke, A.S. Lewis, M.L.O., 2005;K.C. Kiwiel, 2007), but computationally intensive

■ BFGS: traditional workhorse for smooth optimization, worksamazingly well for nonsmooth optimization too, but verylimited convergence theory

Gradient Sampling

Introduction

Gradient Sampling

The GradientSampling Method

With First Phase ofGradient Sampling

With Second Phaseof GradientSampling

The ClarkeSubdifferential

Example

Note that0 ∈ ∂Cf(x) at

x = [1; 1]T

Grad. Samp.: AStabilized SteepestDescent MethodConvergence ofGradient SamplingMethod

ExtensionsSome GradientSampling SuccessStories

Quasi-NewtonMethods

10 / 71

The Gradient Sampling Method

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Fix sample size m ≥ n+ 1, line search parameter β ∈ (0, 1), reductionfactors µ ∈ (0, 1) and θ ∈ (0, 1).

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Initialize sampling radius ǫ > 0, tolerance τ > 0, iterate x.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

■ Repeat (inner loop: gradient sampling with fixed ǫ):

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

◆ Set G = {∇f(x),∇f(x+ ǫu1), . . . ,∇f(x+ ǫum)}, samplingu1, · · · , um uniformly from the unit ball

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

◆ Set g = argmin{||g|| : g ∈ conv(G)}

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

◆ Set g = argmin{||g|| : g ∈ conv(G)}◆ If ‖g ≤ τ , break out of loop.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

◆ Set g = argmin{||g|| : g ∈ conv(G)}◆ If ‖g ≤ τ , break out of loop.◆ Backtracking line search: set d = −g and replace x by x+ td,

with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖

◆ If f is not differentiable at x+ td, replace x+ td by a nearbypoint where f is differentiable.1

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖

■ New phase: set ǫ = µǫ and τ = θτ .

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

11 / 71

Repeat (outer loop)

with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖

■ New phase: set ǫ = µǫ and τ = θτ .

J.V. Burke, A.S. Lewis and M.L.O., SIOPT, 2005.

1Needed in theory, but typically not in practice.

With First Phase of Gradient Sampling

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

12 / 71

f(x)=10*|x2 − x

12| + (1−x

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

With Second Phase of Gradient Sampling

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

13 / 71

f(x)=10*|x2 − x

12| + (1−x

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

The Clarke Subdifferential

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

Assume f : Rn → R is locally Lipschitz, andlet D = {x ∈ R

n : f is differentiable at x}.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

n : f is differentiable at x}.Rademacher’s Theorem: Rn\D has measure zero.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

The Clarke subdifferential of f at x is

∂Cf(x) = conv

x→x,x∈D∇f(x)

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

∂Cf(x) = conv

x→x,x∈D∇f(x)

F.H. Clarke, 1973 (he used the name “generalized gradient”).

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

∂Cf(x) = conv

x→x,x∈D∇f(x)

If f is continuously differentiable at x, then ∂Cf(x) = {∇f(x)}.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

∂Cf(x) = conv

x→x,x∈D∇f(x)

If f is continuously differentiable at x, then ∂Cf(x) = {∇f(x)}.If f is convex, ∂Cf is the subdifferential of convex analysis.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

∂Cf(x) = conv

x→x,x∈D∇f(x)

We say x is Clarke stationary for f if 0 ∈ ∂Cf(x).

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

14 / 71

∂Cf(x) = conv

x→x,x∈D∇f(x)

We say x is Clarke stationary for f if 0 ∈ ∂Cf(x).

Key point: the convex hull of the set G generated by GradientSampling is a surrogate for ∂Cf .

Example

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

Example

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

For x with x2 6= x21, f is differentiable with gradient

∇f(x) = 10 sgn{x2 − x21}[−2x11

[−2(1− x1)

so ∂Cf(x) = {∇f(x)}.

Example

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

∇f(x) = 10 sgn{x2 − x21}[−2x11

[−2(1− x1)

so ∂Cf(x) = {∇f(x)}. For x with x2 = x21, there are twolimiting gradients, namely

[−2x11

[−2(1− x1)

so ∂Cf(x) consists of the convex hull of these two vectors.

Example

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

∇f(x) = 10 sgn{x2 − x21}[−2x11

[−2(1− x1)

so ∂Cf(x) = {∇f(x)}. For x with x2 = x21, there are twolimiting gradients, namely

[−2x11

[−2(1− x1)

so ∂Cf(x) consists of the convex hull of these two vectors.The unique x for which 0 ∈ ∂Cf(x) is x = [1; 1]T , so this is theunique Clarke stationary point of f (it follows that it is theglobal minimizer).

Note that 0 ∈ ∂Cf(x) at x = [1; 1]T

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

16 / 71

f(x)=10*|x2 − x

12| + (1−x

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

Grad. Samp.: A Stabilized Steepest Descent Method

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

Lemma. Let C be a compact convex set and ‖ · ‖ = ‖ · ‖2. Then

−dist(0, C) = min‖d‖≤1

maxg∈C

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

Proof.−dist(0, C) = −min

g∈C‖g‖

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

= − max‖d‖≤1

ming∈C

gT (−d)

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

Note: the distance is nonnegative, and zero iff 0 ∈ C.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

Note: the distance is nonnegative, and zero iff 0 ∈ C.Otherwise, equality is attained by g = ΠC(0), d = −g/|g‖.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

Note: the distance is nonnegative, and zero iff 0 ∈ C.Otherwise, equality is attained by g = ΠC(0), d = −g/|g‖.Ordinary steepest descent: C = {∇f(x)}.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

17 / 71

maxg∈C

g∈C‖g‖

= −ming∈C

max‖d‖≤1

= − max‖d‖≤1

ming∈C

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

Note: the distance is nonnegative, and zero iff 0 ∈ C.Otherwise, equality is attained by g = ΠC(0), d = −g/|g‖.Ordinary steepest descent: C = {∇f(x)}.Gradient sampling: C = conv(G)

= conv({∇f(x),∇f(x+ ǫu1), . . . ,∇f(x+ ǫum)}) .

Convergence of Gradient Sampling Method

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

Theorem. Suppose that f : Rn → R

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

■ is locally Lipschitz

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

■ is locally Lipschitz■ is cont. differentiable on an open full-measure subset of Rn

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

■ has bounded level sets

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

Then, with probability one, f is differentiable at the sampledpoints, the line search always terminates, and if the sequence ofiterates {x} converges to some point x, then, with probability 1

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

■ the inner loop always terminates, so the sequences ofsampling radii {ǫ} and tolerances {τ} converge to zero, and

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

■ x is Clarke stationary for f , i.e., 0 ∈ ∂Cf(x).

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

Drop the assumption that f has bounded level sets. Then, wp 1,either the sequence {f(x)} → −∞, or every cluster point of thesequence of iterates {x} is Clarke stationary.

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

18 / 71

Drop the assumption that f has bounded level sets. Then, wp 1,either the sequence {f(x)} → −∞, or every cluster point of thesequence of iterates {x} is Clarke stationary.

K.C. Kiwiel, SIOPT, 2007.

Extensions

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

19 / 71

A more efficient version: Adaptive Gradient Sampling(F.E. Curtis and X. Que, 2013).

Extensions

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

19 / 71

Problems with Nonsmooth Constraints

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

Extensions

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

19 / 71

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

where f and c1, . . . , cp are locally Lipschitz but may not bedifferentiable at local minimizers.

Extensions

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

19 / 71

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

A successive quadratic programming gradient sampling methodwith convergence theory.

Extensions

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

19 / 71

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

A successive quadratic programming gradient sampling methodwith convergence theory.

F.E. Curtis and M.L.O., SIOPT, 2012.

Some Gradient Sampling Success Stories

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

20 / 71

■ Non-Lipschitz eigenvalue optimization for non-normalmatrices (J.V. Burke, A.S. Lewis and M.L.O., 2002 – )

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

20 / 71

■ Design of fixed-order controllers for linear dynamical systemswith input and output (D. Henrion and M.L.O., 2006, andmany subsequent users of our HIFOO (H-Infinity Fixed OrderOptimization) toolbox)

Introduction

Gradient Sampling

Example

x = [1; 1]T

Quasi-NewtonMethods

20 / 71

■ Design of path planning for robots: avoids “chattering” thatotherwise arises from nonsmoothness (I. Mitchell et al, 2017)

Quasi-Newton Methods

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

BFGSThe BFGS Method(“Full” Version)

BFGS forNonsmoothOptimization

With BFGSExample:Minimizing aProduct ofEigenvalues

BFGS from 10Randomly GeneratedStarting Points

Evolution ofEigenvalues ofA ◦ XEvolution ofEigenvalues of H

Regularity

Partly SmoothFunctionsPartly SmoothFunctions, continued

Same ExampleAgain

Relation of Partial

21 / 71

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

22 / 71

W. Davidon, a physicist at Argonne, had the breakthrough ideain 1959: since it’s too expensive to compute and factor theHessian ∇2f(x) at every iteration, update an approximation toits inverse using information from gradient differences, andmultiply this onto the negative gradient to approximateNewton’s method.

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

22 / 71

Each inverse Hessian approximation differs from the previous oneby a rank-two correction.

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

22 / 71

Ahead of its time: the paper was rejected by the physicsjournals, but published 30 years later in the first issue of SIAM J.Optimization.

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

22 / 71

Ahead of its time: the paper was rejected by the physicsjournals, but published 30 years later in the first issue of SIAM J.Optimization.

Davidon was a well known active anti-war protester during theVietnam War. In December 2013, it was revealed that he wasthe mastermind behind the break-in at the FBI office in Media,PA, on March 8, 1971, during the Muhammad Ali - Joe Frazierworld heavyweight boxing championship.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

23 / 71

In 1963, R. Fletcher and M.J.D. Powell improved Davidon’smethod and established convergence for convex quadraticfunctions.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

23 / 71

They applied it to solve problems in 100 variables: a lot at thetime.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

23 / 71

The method became known as the DFP method.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

23 / 71

The method became known as the DFP method.

Davidon, Fletcher and Powell all died during 2013–2016.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

24 / 71

In 1970, C.G. Broyden, R. Fletcher, D. Goldfarb and D. Shannoall independently proposed the BFGS method, which is a kind ofdual of the DFP method. It was soon recognized that this was aremarkably effective method for smooth optimization.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

24 / 71

In 1973, C.G. Broyden, J.E. Dennis and J.J. More proved genericlocal superlinear convergence of BFGS and DFP and otherquasi-Newton methods.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

24 / 71

In 1976, M.J.D. Powell established convergence of BFGS with aninexact Armijo-Wolfe line search for a general class of smoothconvex functions for BFGS. In 1987, this was extended byR.H. Byrd, J. Nocedal and Y.-X. Yuan to include the whole“Broyden” class of methods interpolating BFGS and DFP:except for the DFP end point.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

24 / 71

In 1976, M.J.D. Powell established convergence of BFGS with aninexact Armijo-Wolfe line search for a general class of smoothconvex functions for BFGS. In 1987, this was extended byR.H. Byrd, J. Nocedal and Y.-X. Yuan to include the whole“Broyden” class of methods interpolating BFGS and DFP:except for the DFP end point.

Pathological counterexamples to convergence in the smooth,nonconvex case are known to exist (Y.-H. Dai, 2002, 2013;W. Mascarenhas 2004), but it is widely accepted that themethod works well in practice in the smooth, nonconvex case.

The BFGS Method (“Full” Version)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Initialize iterate x and positive-definite symmetric matrix H(which is supposed to approximate the inverse Hessian of f)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

■ Set d = −H∇f(x).

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search■ Set s = td, y = ∇f(x+ td)−∇f(x)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search■ Set s = td, y = ∇f(x+ td)−∇f(x)■ Replace x by x+ td

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search■ Set s = td, y = ∇f(x+ td)−∇f(x)■ Replace x by x+ td■ Replace H by V HV T + 1

sT yssT , where V = I − 1

sT ysyT

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

sT ysyT

Note that H can be computed in O(n2) operations since V is arank one perturbation of the identity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

25 / 71

Repeat

sT ysyT

Note that H can be computed in O(n2) operations since V is arank one perturbation of the identityThe Wolfe condition guarantees that sT y > 0 and hence thatthe new H is positive definite.

BFGS for Nonsmooth Optimization

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

In 1982, C. Lemarechal observed that quasi-Newton methods can beeffective for nonsmooth optimization, but dismissed them as there wasno theory behind them and no good way to terminate them.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

Otherwise, there is not much in the literature on the subject untilA.S. Lewis and M.L.O. (Math. Prog., 2013): we address both issues indetail, but our convergence results are limited to special cases.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

Key point: use an Armijo-Wolfe line search. Do not insist on reducingthe magnitude of the directional derivative along the line!

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

In the nonsmooth case, BFGS builds a very ill-conditioned inverse“Hessian” approximation, with some tiny eigenvalues converging tozero, corresponding to “infinitely large” curvature in the directionsdefined by the associated eigenvectors.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

Remarkably, the condition number of the inverse Hessianapproximation typically reaches 1016 before the method breaks down.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

We have never seen convergence to non-stationary points that cannotbe explained by numerical difficulties.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

26 / 71

We have never seen convergence to non-stationary points that cannotbe explained by numerical difficulties.

Convergence rate of BFGS is typically linear (not superlinear) in thenonsmooth case.

With BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

27 / 71

f(x)=10*|x2 − x

12| + (1−x

steepest descent, grad sampling and BFGS iterates

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

contours of fstarting pointoptimal pointsteepest descentgrad samp (1st phase)grad samp (2nd phase)bfgs

Example: Minimizing a Product of Eigenvalues

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

28 / 71

Let SN denote the space of real symmetric N ×N matrices, and

λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

denote the eigenvalues of X ∈ SN .

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

28 / 71

λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

denote the eigenvalues of X ∈ SN . We wish to minimize

f(X) = log

N/2∏

λi(A ◦X)

where A ∈ SN is fixed and ◦ is the Hadamard (componentwise)matrix product, subject to the constraints that X is positivesemidefinite and has diagonal entries equal to 1.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

28 / 71

λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

f(X) = log

N/2∏

λi(A ◦X)

If we replace∏

we would have a semidefinite program.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

28 / 71

λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

f(X) = log

N/2∏

λi(A ◦X)

If we replace∏

Since f is not convex, may as well replace X by Y Y T whereY ∈ R

N×N : eliminates psd constraint, and then also easy toeliminate diagonal constraint.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

28 / 71

λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

f(X) = log

N/2∏

λi(A ◦X)

If we replace∏

Since f is not convex, may as well replace X by Y Y T whereY ∈ R

N×N : eliminates psd constraint, and then also easy toeliminate diagonal constraint.

Application: entropy minimization in an environmentalapplication (K.M. Anstreicher and J. Lee, 2004)

BFGS from 10 Randomly Generated Starting Points

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

29 / 71

0 200 400 600 800 1000 1200 140010

10−10

10−5

iteration

f − f op

Log eigenvalue product, N=20, n=400, fopt

= −4.37938e+000

f − fopt, where fopt is least value of f found over all runs

Evolution of Eigenvalues of A ◦X

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

30 / 71

0 200 400 600 800 1000 1200 14000.2

= −4.37938e+000

iteration

Evolution of Eigenvalues of A ◦X

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

30 / 71

0 200 400 600 800 1000 1200 14000.2

= −4.37938e+000

iteration

Note that λ6(X), . . . , λ14(X) coalesce

Evolution of Eigenvalues of H

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

31 / 71

0 200 400 600 800 1000 1200 140010

10−10

10−5

= −4.37938e+000

iteration

Evolution of Eigenvalues of H

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

31 / 71

0 200 400 600 800 1000 1200 140010

10−10

10−5

= −4.37938e+000

iteration

44 eigenvalues of H converge to zero...why???

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

32 / 71

A locally Lipschitz, directionally differentiable function f isregular (Clarke 1970s) near a point x when its directionalderivative f ′( · ; d) is upper semicontinuous there for every fixeddirection d.

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

32 / 71

In this case 0 ∈ ∂Cf(x) is equivalent to the first-order optimalitycondition f ′(x, d) ≥ 0 for all directions d.

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

32 / 71

■ All convex functions are regular

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

32 / 71

■ All convex functions are regular■ All smooth functions are regular

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

32 / 71

■ All convex functions are regular■ All smooth functions are regular■ Nonsmooth concave functions are not regular

Example: f(x) = −|x|

Note: this is a somewhat simpler definition of regularity than theone in Lecture 12, but it is less precise: it defines regularity in aneighborhood, not at a point.

Partly Smooth Functions

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

A regular function f is partly smooth at x relative to a manifoldM containing x (A.S. Lewis 2003) if

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

■ its restriction to M is twice continuously differentiable near x

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

■ its restriction to M is twice continuously differentiable near x■ the Clarke subdifferential ∂f is continuous on M near x

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

■ its restriction to M is twice continuously differentiable near x■ the Clarke subdifferential ∂f is continuous on M near x■ par ∂f(x), the subspace parallel to the affine hull of the

subdifferential of f at x, is exactly the subspace normal toM at x.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

We refer to par ∂f(x) as the V -space for f at x (with respect toM), and to its orthogonal complement, the subspace tangent toM at x, as the U -space for f at x.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

When we refer to the V -space and U -space without reference toa point x, we mean at a minimizer.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

33 / 71

When we refer to the V -space and U -space without reference toa point x, we mean at a minimizer.

For nonzero y in the V -space, the mapping t 7→ f(x+ ty) isnecessarily nonsmooth at t = 0, while for nonzero y in theU -space, t 7→ f(x+ ty) is differentiable at t = 0 as long as f islocally Lipschitz.

Partly Smooth Functions, continued

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

34 / 71

Example: f(x) = 10|x2 − x21|+ (1− x1)2.

Question: What is M and what are the U and V spaces at theminimizer?

Partly Smooth Functions, continued

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

34 / 71

Example: f(x) = 10|x2 − x21|+ (1− x1)2.

Question: What is M and what are the U and V spaces at theminimizer?

Example: f(x) = ‖x‖2.Question: What is M and what are the U and V spaces at theminimizer?

Same Example Again

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

35 / 71

f(x)=10*|x2 − x

12| + (1−x

steepest descent, grad sampling and BFGS iterates

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

contours of fstarting pointoptimal pointsteepest descentgrad samp (1st phase)grad samp (2nd phase)bfgs

Relation of Partial Smoothness to Earlier Work

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

36 / 71

Partial smoothness is closely related to earlier work of J.V. Burkeand J.J. More (1990,1994) and S.J. Wright (1993) onidentification of constraint structure by algorithms.

Relation of Partial Smoothness to Earlier Work

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

36 / 71

Partial smoothness is closely related to earlier work of J.V. Burkeand J.J. More (1990,1994) and S.J. Wright (1993) onidentification of constraint structure by algorithms.

When f is convex, the partly smooth nomenclature is consistentwith the usage of V -space and U -space by C. Lemarechal,F. Oustry and C. Sagastizabal (2000), but partial smoothnessdoes not imply convexity and convexity does not imply partialsmoothness.

Why Did 44 Eigenvalues of H Converge to Zero?

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

37 / 71

The eigenvalue product is regular and also partly smooth (in the senseof A.S. Lewis, 2003) with respect to the manifold of matrices with aneigenvalue with given multiplicity. This implies that tangent to thismanifold (preserving the multiplicity to first-order) the function issmooth (“U -shaped”) and normal to it, the function is nonsmooth

(“V -shaped”).

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

37 / 71

(“V -shaped”).

Recall that at the computed minimizer,

λ6(A ◦X) ≈ . . . ≈ λ14(A ◦X).

Matrix theory says that imposing multiplicity m on an eigenvalue a

matrix ∈ SN is m(m+1)2 − 1 conditions, or 44 when m = 9, so the

dimension of the V -space at this minimizer is 44.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

37 / 71

(“V -shaped”).

λ6(A ◦X) ≈ . . . ≈ λ14(A ◦X).

Tiny eigenvalues of H correspond to huge curvature, whichcorresponds to V -space directions.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

37 / 71

(“V -shaped”).

λ6(A ◦X) ≈ . . . ≈ λ14(A ◦X).

Tiny eigenvalues of H correspond to huge curvature, whichcorresponds to V -space directions.

Thus BFGS automatically detected the U and V space partitioningwithout knowing anything about the mathematical structure of f !

Variation of f from Minimizer, along EigVecs of H

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

38 / 71

−10 −5 0 5 100

f(x op

− f op

= −4.37938e+000

w is eigvector for eigvalue 10 of final Hw is eigvector for eigvalue 20 of final Hw is eigvector for eigvalue 30 of final Hw is eigvector for eigvalue 40 of final Hw is eigvector for eigvalue 50 of final Hw is eigvector for eigvalue 60 of final H

Eigenvalues of H numbered smallest to largest

BFGS Theory for Special Nonsmooth Functions

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

39 / 71

Convergence results for BFGS with Armijo-Wolfe line searchwhen f is nonsmooth are limited to very special cases.

■ f(x) = |x| (one variable!): sequence generated converging to0 is related to a certain binary expansion of the startingpoint (A.S. Lewis and M.L.O., 2013)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

39 / 71

■ f(x) = |x1|+ x2: f(x) ↓ −∞ (A.S. Lewis and ShanshanZhang, 2015)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

39 / 71

■ f(x) = |x1|+∑n

i=2 xi: eventually a direction is identified onwhich f is unbounded below (Yuchen Xie and A. Waechter,2017)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

39 / 71

■ f(x) = |x1|+∑n

■ f(x) =√∑n

i=1 x2i : iterates converge to [0, . . . , 0] (Jiayi Guo

and A.S. Lewis, 2017) (proof based on Powell (1976))

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

39 / 71

■ f(x) = |x1|+∑n

■ f(x) =√∑n

i=1 x2i : iterates converge to [0, . . . , 0] (Jiayi Guo

and A.S. Lewis, 2017) (proof based on Powell (1976))

■ f(x) = |x1|+ x22: remains open!

Challenge: General Nonsmooth Case

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

Assume f is locally Lipschitz with bounded level sets and issemi-algebraic (its graph is a finite union of sets each defined bya finite list of polynomial inequalities)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

Assume the initial x and H are generated randomly (e.g. fromnormal and Wishart distributions)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

Prove or disprove that the following hold with probability one:

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

1. BFGS generates an infinite sequence {x} with fdifferentiable at all iterates

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

2. Any cluster point x is Clarke stationary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

2. Any cluster point x is Clarke stationary3. The sequence of function values generated (including all of

the line search iterates) converges to f(x) R-linearly

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

40 / 71

2. Any cluster point x is Clarke stationary3. The sequence of function values generated (including all of

the line search iterates) converges to f(x) R-linearly4. If {x} converges to x where f is “partly smooth” w.r.t. a

manifold M then the subspace defined by the eigenvectorscorresponding to eigenvalues of H converging to zeroconverges to the “V -space” of f w.r.t. M at x

Some BFGS Nonsmooth Success Stories

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

41 / 71

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

41 / 71

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

41 / 71

■ Shape optimization for spectral functions ofDirichlet-Laplacian operators (B. Osting, 2010)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

41 / 71

■ Condition metric optimization (P. Boito and J. Dedieu, 2010)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

41 / 71

■ Condition metric optimization (P. Boito and J. Dedieu, 2010)

Software is available: HANSO

Extensions of BFGS for Nonsmooth Optimization

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

42 / 71

A combined BFGS-Gradient Sampling method with convergencetheory (F.E. Curtis and X. Que, 2015)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

42 / 71

Constrained Problems

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

42 / 71

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

A successive quadratic programming (SQP) BFGS methodapplied to challenging problems in static-output-feedback controldesign (F.E. Curtis, T. Mitchell and M.L.O., 2015).

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

42 / 71

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

Although there are no theoretical results, it is much moreefficient and effective than the SQP Gradient Sampling methodwhich does have convergence results.

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

Regularity

Same ExampleAgain

Relation of Partial

42 / 71

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

Although there are no theoretical results, it is much moreefficient and effective than the SQP Gradient Sampling methodwhich does have convergence results.

Software is available: GRANSO

A Difficult Nonconvex Problem

from Nesterov

Introduction

Gradient Sampling

Quasi-NewtonMethods

A DifficultNonconvex Problemfrom NesterovAn Aside:ChebyshevPolynomials

Plots of ChebyshevPolynomials

Nesterov’sChebyshev-RosenbrockFunctionsWhy BFGS Takes SoMany Iterations toMinimize N2

First NonsmoothVariant ofNesterov’s FunctionSecond NonsmoothVariant ofNesterov’s FunctionContour Plots of theNonsmooth Variantsfor n = 2Properties of theSecond NonsmoothVariant N1

Behavior of BFGSon the SecondNonsmooth Variant

43 / 71

An Aside: Chebyshev Polynomials

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

A sequence of orthogonal polynomials defined on [−1, 1] by

T0(x) = 1, T1(x) = x, Tn+1(x) = 2xTn(x)− Tn−1(x).

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

So T2(x) = 2x2 − 1,

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.

Important properties that can be proved easily include

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.

■ Tn(x) = cos(n cos−1(x))

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.

■ Tm(Tn(x)) = Tmn(x)

Introduction

Gradient Sampling

Quasi-NewtonMethods

44 / 71

So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.

■ Tm(Tn(x)) = Tmn(x)

∫ 1−1

1√1−x2

Ti(x)Tj(x)dx = 0 if i 6= j

Plots of Chebyshev Polynomials

Introduction

Gradient Sampling

Quasi-NewtonMethods

45 / 71

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

Left: Plots of T0(x), . . . , T4(x) Right: Plot of T8(x).

Plots of Chebyshev Polynomials

Introduction

Gradient Sampling

Quasi-NewtonMethods

45 / 71

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

Left: Plots of T0(x), . . . , T4(x) Right: Plot of T8(x).Question: How many extrema does Tn(x) have in [−1, 1]?

Nesterov’s Chebyshev-Rosenbrock Functions

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Consider the function

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

The unique minimizer is x∗ = [1, 1, . . . , 1]T with Np(x∗) = 0.

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

Define x = [−1, 1, 1, . . . , 1]T with Np(x) = 1 and the manifold

MN = {x : xi+1 = 2x2i − 1, i = 1, . . . , n− 1}

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

MN = {x : xi+1 = 2x2i − 1, i = 1, . . . , n− 1}For x ∈ MN , e.g. x = x∗ or x = x, the 2nd term of Np is zero.Starting at x, BFGS needs to approximately follow MN to reachx∗ (unless it “gets lucky”).

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

When p = 2: N2 is smooth but not convex. Starting at x:

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

■ n = 5: BFGS needs 370 iterations to reduce N2 below 10−15

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

■ n = 10: needs ∼ 50,000 iterations to reduce N2 below 10−15

even though N2 is smooth!

Introduction

Gradient Sampling

Quasi-NewtonMethods

46 / 71

Np(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|p, where p ≥ 1

■ n = 10: needs ∼ 50,000 iterations to reduce N2 below 10−15

even though N2 is smooth! . . . In the last few iterations, weobserve superlinear convergence!

Why BFGS Takes So Many Iterations to Minimize N2

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

Let Ti(x) denote the ith Chebyshev polynomial. For x ∈ MN ,

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

To move from x to x∗ along the manifold MN exactly requires

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

■ x1 to change from −1 to 1

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

■ x1 to change from −1 to 1■ x2 = 2x21 − 1 to trace the graph of T2(x1) on [−1, 1]

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

■ x1 to change from −1 to 1■ x2 = 2x21 − 1 to trace the graph of T2(x1) on [−1, 1]■ x3 = T2(T2(x)) to trace the graph of T4(x1) on [−1, 1]

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

■ x1 to change from −1 to 1■ x2 = 2x21 − 1 to trace the graph of T2(x1) on [−1, 1]■ x3 = T2(T2(x)) to trace the graph of T4(x1) on [−1, 1]■ xn = T2n−1(x) to trace the graph of T2n−1(x1) on [−1, 1]

which has 2n−1 − 1 extrema in (−1, 1).

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

which has 2n−1 − 1 extrema in (−1, 1).Even though BFGS will not track the manifold MN exactly, itwill follow it approximately. So, since the manifold is highlyoscillatory, BFGS must take relatively short steps to obtainreduction in N2 in the line search, and hence many iterations!

Introduction

Gradient Sampling

Quasi-NewtonMethods

47 / 71

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

which has 2n−1 − 1 extrema in (−1, 1).Even though BFGS will not track the manifold MN exactly, itwill follow it approximately. So, since the manifold is highlyoscillatory, BFGS must take relatively short steps to obtainreduction in N2 in the line search, and hence many iterations!

Newton’s method is not much faster, although it convergesquadratically at the end.

First Nonsmooth Variant of Nesterov’s Function

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

N1 is nonsmooth (though locally Lipschitz) as well asnonconvex. The second term is still zero on the manifold MN ,but N1 is not differentiable on MN .

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

However, N1 is regular at x ∈ MN and partly smooth at x w.r.t.MN .

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

We cannot initialize BFGS at x, so starting at normallydistributed random points:

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

■ n = 5: BFGS reduces N1 only to about 10−2 in 10,000iterations

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

■ n = 10: BFGS reduces N1 only to about 5× 10−2 in 10,000iterations

Introduction

Gradient Sampling

Quasi-NewtonMethods

48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

|xi+1 − 2x2i + 1|

■ n = 10: BFGS reduces N1 only to about 5× 10−2 in 10,000iterations

The method appears to be converging, very slowly, but may behaving numerical difficulties.

Second Nonsmooth Variant of Nesterov’s Function

Introduction

Gradient Sampling

Quasi-NewtonMethods

49 / 71

N1(x) =1

4|x1 − 1|+

n−1∑

|xi+1 − 2|xi|+ 1|.

Again, the unique global minimizer is x∗. The second term iszero on the set

S = {x : xi+1 = 2|xi| − 1, i = 1, . . . , n− 1}

but S is not a manifold: it has “corners”.

Contour Plots of the Nonsmooth Variants for n = 2

Introduction

Gradient Sampling

Quasi-NewtonMethods

50 / 71

Nesterov−Chebyshev−Rosenbrock, first variant

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

2Nesterov−Chebyshev−Rosenbrock, second variant

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

Contour plots of nonsmooth Chebyshev-Rosenbrock functions N1

(left) and N1 (right), with n = 2, with iterates generated byBFGS initialized at 7 different randomly generated points.

Contour Plots of the Nonsmooth Variants for n = 2

Introduction

Gradient Sampling

Quasi-NewtonMethods

50 / 71

Nesterov−Chebyshev−Rosenbrock, first variant

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

2Nesterov−Chebyshev−Rosenbrock, second variant

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−0.5

Contour plots of nonsmooth Chebyshev-Rosenbrock functions N1

(left) and N1 (right), with n = 2, with iterates generated byBFGS initialized at 7 different randomly generated points.On the left, always get convergence to x∗ = [1, 1]T . On theright, most runs converge to [1, 1] but some go to x = [0,−1]T .

Properties of the Second Nonsmooth Variant N1

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

When n = 2, the point x = [0,−1]T is Clarke stationary for thesecond nonsmooth variant N1. We can see this because zero isin the convex hull of the gradient limits for N1 at the point x.

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

However, x = [0,−1]T is not a local minimizer, becaused = [1, 2]T is a direction of linear descent: N ′

1(x, d) < 0.

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

1(x, d) < 0.

These two properties mean that N1 is not regular at [0,−1]T .

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

1(x, d) < 0.

In fact, for n ≥ 2:

■ N1 has 2n−1 Clarke stationary points

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

1(x, d) < 0.

■ N1 has 2n−1 Clarke stationary points■ the only local minimizer is the global minimizer x∗

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

1(x, d) < 0.

■ x∗ is the only stationary point in the sense of Mordukhovich(i.e., with 0 ∈ ∂N1(x) where we defined ∂ in Lecture 12)(see also Rockafellar and Wets, Variational Analysis, 1998).

(M. Gurbuzbalaban and M.L.O., 2012)

Introduction

Gradient Sampling

Quasi-NewtonMethods

51 / 71

1(x, d) < 0.

■ x∗ is the only stationary point in the sense of Mordukhovich(i.e., with 0 ∈ ∂N1(x) where we defined ∂ in Lecture 12)(see also Rockafellar and Wets, Variational Analysis, 1998).

(M. Gurbuzbalaban and M.L.O., 2012)

Furthermore, starting from enough randomly generated startingpoints, BFGS finds all 2n−1 Clarke stationary points!

Behavior of BFGS on the Second Nonsmooth Variant

Introduction

Gradient Sampling

Quasi-NewtonMethods

52 / 71

0 200 400 600 800 10000

0.5Nesterov−Chebyshev−Rosenbrock, n=5

different starting points

0 200 400 600 800 10000

0.5Nesterov−Chebyshev−Rosenbrock, n=6

different starting points

Left: sorted final values of N1 for 1000 randomly generatedstarting points, when n = 5: BFGS finds all 16 Clarke stationarypoints. Right: same with n = 6: BFGS finds all 32 Clarkestationary points.

Convergence to Non-Locally-Minimizing Points

Introduction

Gradient Sampling

Quasi-NewtonMethods

53 / 71

When f is smooth, convergence of methods such as BFGS tonon-locally-minimizing stationary points or local maxima ispossible but not likely, because of the line search, and suchconvergence will not be stable under perturbation.

Introduction

Gradient Sampling

Quasi-NewtonMethods

53 / 71

However, this kind of convergence is what we are seeing for thenon-regular, non-smooth Nesterov Chebyshev-Rosenbrockexample, and it is stable under perturbation. The same behavioroccurs for gradient sampling or bundle methods.

Introduction

Gradient Sampling

Quasi-NewtonMethods

53 / 71

Kiwiel (private communication): the Nesterov example is the firsthe had seen which causes his bundle code to have this behavior.

Introduction

Gradient Sampling

Quasi-NewtonMethods

53 / 71

Kiwiel (private communication): the Nesterov example is the firsthe had seen which causes his bundle code to have this behavior.

Nonetheless, we don’t know whether, in exact arithmetic, themethods would actually generate sequences converging to thenonminimizing Clarke stationary points. Experiments by Kaku(2011) suggest that the higher the precision used, the more likelyBFGS is to eventually move away from such a point.

Limited Memory Methods

Introduction

Gradient Sampling

Quasi-NewtonMethods

Limited MemoryMethodsLimited MemoryBFGSLimited MemoryBFGS on theEigenvalue Product

A More BasicExample

Smooth, Convex:nA = 200, nB =0, nR = 1

Nonsmooth, Convex:nA = 200, nB =10, nR = 1

Nonsmooth,Nonconvex:nA = 200, nB =10, nR = 5

A NonsmoothConvex Function,Unbounded BelowL-BFGS-1 vs.Gradient DescentConvergence of theL-BFGS-1 Search

54 / 71

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

55 / 71

“Full” BFGS requires storing an n× n matrix and doingmatrix-vector multiplies, which is not possible when n is large.

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

55 / 71

In the 1980s, J. Nocedal and others developed a “limitedmemory” version of BFGS, with O(n) space and timerequirements, which is very widely used for minimizing smoothfunctions in many variables. At the kth iteration, it applies onlythe most recent m rank-two updates, defined by

(sj , yj), j = k −m, . . . , k − 1

to an initial inverse Hessian approximation H(k)0 .

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

55 / 71

(sj , yj), j = k −m, . . . , k − 1

There are two variants: with “scaling” (H(k)0 =

sTk−1yk−1

yTk−1yk−1

I) and

without scaling (H(k)0 = I).

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

55 / 71

(sj , yj), j = k −m, . . . , k − 1

sTk−1yk−1

yTk−1yk−1

I) and

The convergence rate of limited memory BFGS is linear, notsuperlinear, on smooth problems.

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

55 / 71

(sj , yj), j = k −m, . . . , k − 1

sTk−1yk−1

yTk−1yk−1

I) and

The convergence rate of limited memory BFGS is linear, notsuperlinear, on smooth problems.

Question: how effective is it on nonsmooth problems?

Limited Memory BFGS on the Eigenvalue Product

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

56 / 71

0 10 20 30 40 50

10−5

10−4

10−3

10−2

10−1

Log eig prod, N=20, r=20, K=10, n=400, maxit = 400

number of vectors k

f−f op

Lim Mem BFGS (scaling)Lim Mem BFGS (no scaling)full BFGS (scaling once)full BFGS (no scaling)

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

56 / 71

0 10 20 30 40 50

10−5

10−4

10−3

10−2

10−1

number of vectors k

f−f op

Limited Memory is not nearly as good as full BFGS.

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

56 / 71

0 10 20 30 40 50

10−5

10−4

10−3

10−2

10−1

number of vectors k

f−f op

Limited Memory is not nearly as good as full BFGS.

No significant improvement when k reaches 44.

A More Basic Example

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

57 / 71

Let x = [y; z;w] ∈ RnA+nB+nR and consider the test function

f(x) = (y − e)TA(y − e) + {(z − e)TB(z − e)}1/2 +R1(w)

where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

57 / 71

where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].

The first term is quadratic, the second is nonsmooth but convex,and the third is a nonsmooth, nonconvex Rosenbrock function.

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

57 / 71

where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].

The optimal value is 0, with x = e. The function f is partlysmooth and the dimension of the V-space is nB + nR − 1.

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

57 / 71

where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].

Set A = XXT where xij are normally distributed, with conditionnumber about 106 when nA = 200. Similarly B with nB < nA.

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

57 / 71

where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].

Set A = XXT where xij are normally distributed, with conditionnumber about 106 when nA = 200. Similarly B with nB < nA.

Besides limited memory BFGS and full BFGS, we also comparelimited memory Gradient Sampling, where we sample k ≪ ngradients per iteration.

Smooth, Convex: nA = 200, nB = 0, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

58 / 71

0 5 10 15 2010

10−6

10−5

nA=200, nB=0, nR=1, maxit = 201

number of vectors k

Grad Samp 1e−02, 1e−04Lim Mem BFGS (scaling)Lim Mem BFGS (no scaling)full BFGS (scaling once)full BFGS (no scaling)

Smooth, Convex: nA = 200, nB = 0, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

58 / 71

0 5 10 15 2010

10−6

10−5

nA=200, nB=0, nR=1, maxit = 201

number of vectors k

LM-BFGS with scaling even better than full BFGS

Nonsmooth, Convex: nA = 200, nB = 10, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

59 / 71

0 5 10 15 2010

10−5

10−4

10−3

10−2

nA=200, nB=10, nR=1, maxit = 211

number of vectors k

Nonsmooth, Convex: nA = 200, nB = 10, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

59 / 71

0 5 10 15 2010

10−5

10−4

10−3

10−2

nA=200, nB=10, nR=1, maxit = 211

number of vectors k

LM-BFGS much worse than full BFGS

Nonsmooth, Nonconvex: nA = 200, nB = 10, nR = 5

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

60 / 71

0 5 10 15 2010

10−3

10−2

10−1

nA=200, nB=10, nR=5, maxit = 215

number of vectors k

Nonsmooth, Nonconvex: nA = 200, nB = 10, nR = 5

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

60 / 71

0 5 10 15 2010

10−3

10−2

10−1

nA=200, nB=10, nR=5, maxit = 215

number of vectors k

LM-BFGS with scaling even worse than LM-Grad-Samp

A Nonsmooth Convex Function, Unbounded Below

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

61 / 71

Let’s reconsiderf(x) = a|x1|+ x2

with a ≥ 1.

-5-10-10

Turns out that L-BFGS-1 (saving just one update) with scalingfails for smaller values of a than the critical value beyond whichGradient Descent fails!

L-BFGS-1 vs. Gradient Descent

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

62 / 71

Red: path of L-BFGS-1 with scaling, converges to non-stationarypoint.Blue: path of the gradient method with same Armijo-Wolfe linesearch, generates f(x) ↓ −∞.

u-8 -6 -4 -2 0 2 4 6 8 10

f(u, v) = 3|u|+ v. x0 = (8.284; 2.177), c1=0.05, τ =-0.056

LBFGS-1Gradient method

Convergence of the L-BFGS-1 Search Direction

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

63 / 71

Theorem. Let d(k) be the search direction generated byL-BFGS-1 with scaling applied to f(x) = a|x1|+

∑ni=2 xi using

an Armijo-Wolfe line search. If√

4(n− 1) ≤ a, then|d(k)|||d(k)||

converges to some constant direction d. Furthermore, if

a(a+√

a2 − 3(n− 1)) > (1

c1− 1)(n− 1),

where c1 is the Armijo parameter, then the iterates x(k) convergeto a non-stationary point.

Azam Asl, 2018.

Experiment, with n = 2 and a =√3

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

64 / 71

In practice we observe that√

3(n− 1) ≤ a suffices for themethod to fail, which is a weaker condition than the previousone. Below with n = 2 and a =

√3 the method fails:

u-8 -6 -4 -2 0 2 4 6 8 10

f(u, v) = 1.7321|u|+ v. x0 = (8.284; 2.177), c1=0.05, τ =-0.27

Experiment: slightly smaller a

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

65 / 71

But if we set a =√3− 0.001, it succeeds “at the last minute”.

u-8 -6 -4 -2 0 2 4 6 8 10

f(u, v) = 1.7311|u|+ v. x0 = (8.284; 2.177), c1=0.05, τ =-0.27

Experiments: Top, scaling on; Bottom, scaling off

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

66 / 71

n = 30,√

3(n− 1) = 9.327

N=30, f(X) = a|x(1)|+ Σi=2N x(i), c

1=0.05, (3(N-1))0.5 = 9.33, nrand = 5000

a9.315 9.32 9.325 9.33 9.335 9.34

Limited Effectiveness of Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

67 / 71

We have observed that that addition of nonsmoothness to aproblem, convex or nonconvex, creates great difficulties forLimited Memory BFGS, with and without scaling, even when thedimension of the V -space is less than the size of the memory.

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

67 / 71

Azam Asl’s result establishes failure of L-BFGS-1 for a specific fwhen scaling is on; no such result is proved yet when scaling isoff.

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

67 / 71

Azam Asl’s result establishes failure of L-BFGS-1 for a specific fwhen scaling is on; no such result is proved yet when scaling isoff.

We have also investigated Limited Memory Gradient Samplingwhich does not work well either.

Other Ideas for Large Scale Nonsmooth Optimization

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

68 / 71

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

68 / 71

■ Exploit structure! Lots of work on this has been done, e.g.using proximal point methods or ADMM (AlternatingDirection Method of Multipliers)

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

68 / 71

■ Smoothing! Lots of work on this has been done too, mostnotably by Yu. Nesterov

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

68 / 71

■ Automatic Differentiation (AD): (A. Griewank et. al.)

Introduction

Gradient Sampling

Quasi-NewtonMethods

A More BasicExample

68 / 71

■ Automatic Differentiation (AD): (A. Griewank et. al.)

■ Stochastic Subgradient Method (D. Davis andD. Drusvyatskiy, 2018)

Concluding Remarks

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

69 / 71

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

70 / 71

Gradient descent frequently fails on nonsmooth problems.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

70 / 71

Gradient Sampling is a simple method for nonsmooth, nonconvexoptimization for which a convergence theory is known, but it istoo expensive to use in most applications.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

70 / 71

BFGS — the full version — is remarkably effective onnonsmooth problems, but little theory is known.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

70 / 71

Limited Memory BFGS is not so effective on nonsmoothproblems.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

70 / 71

Diabolical nonconvex problems such as Nesterov’sChebyshev-Rosenbrock problems can be very difficult, especiallyin the nonsmooth case.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

70 / 71

Diabolical nonconvex problems such as Nesterov’sChebyshev-Rosenbrock problems can be very difficult, especiallyin the nonsmooth case.

Our software, HANSO and GRANSO, is available (unconstrainedand constrained) along with HIFOO (H-infinity fixed orderoptimization) for controller design, which has been usedsuccessfully in many applications.

Our Papers

Introduction

Gradient Sampling

Quasi-NewtonMethods

Concluding Remarks

Summary

Our Papers

71 / 71

J. V. Burke, A. S. Lewis and M. L. Overton, A Robust GradientSampling Method for Nonsmooth, Nonconvex Optimization, SIAMJ. Optimization, 2005

M. Gurbuzbalaban and M. L. Overton, On Nesterov’s NonsmoothChebyshev-Rosenbrock Functions, SIAM J. Optimization, 2012

A. S. Lewis and M. L. Overton, Nonsmooth Optimization viaQuasi-Newton Methods, Math. Programming, 2013

F. E. Curtis, T. Mitchell and M.L. Overton, A BFGS-SQP Method forNonsmooth, Nonconvex, Constrained Optimization and its Evaluationusing Relative Minimization Profiles, Optimization Methods and

Software, 2016

A. Asl and M. L. Overton, Analysis of the Gradient Method with anArmijo-Wolfe Line Search on a Class of Nonsmooth Convex Functions,arXiv, 2017

J. V. Burke, F. E. Curtis, A. S. Lewis, M. L. Overton and L. Simoes,Gradient Sampling Methods for Nonsmooth Optimization, arXiv, 2018

Papers, software are available at www.cs.nyu.edu/overton.

Nonsmooth, Nonconvex Optimization€¦ · Convex and Nonsmooth Optimization Class, Spring 2018,...

Documents

Nonconvex Optimization for Communication Systems · Nonconvex Optimization for Communication Systems Mung Chiang Electrical Engineering Department Princeton University, Princeton,

A feasible second order bundle algorithm for nonsmooth nonconvex optimization problems ... · 2014. 10. 1. · Linearly constrained nonsmooth optimization. There exist diﬀerent

Nonconvex Optimization Methods: Iteration Complexity and

On Nesterov’s Nonsmooth Chebyshev-Rosenbrock Functionsaspremon/Houches/talks/lesHouches... · 2016. 3. 6. · Nonsmooth Optimization With BFGS Some Nonsmooth Analysis Nesterov’s

Introduction to nonconvex optimization

Robust Control meets nonsmooth optimization

Nonsmooth Analysis and Optimization

Adaptive Methods for Nonconvex Optimization

Convex Analysis and Nonsmooth Optimization

Smoothing methods for nonsmooth, nonconvex minimization · Smoothing methods for nonsmooth, nonconvex minimization 73 Deﬁnition 1 Let f: Rn → R be a continuous function. We call

Lecture 1. Essentials of numerical nonsmooth optimization

Hidden Convexity in Nonconvex Quadratic Optimization

Structured Nonconvex and Nonsmooth Optimization ... · Algorithms and Iteration Complexity Analysis Bo Jiang Tianyi Lin y Shiqian Ma z Shuzhong Zhang x November 13, 2017 Abstract

Nonsmooth Optimization: Bundle Methodssalserver.org.aalto.fi/vanhat_sivut/tutkijakoulu/Vienna...Introduction Nonsmooth Optimization Standard Bundle MethodThe Goal of Research Cause

Nonsmooth optimization: Subgradient Method, Proximal ...yuxiangw/classes/CS292A... · Nonsmooth optimization: Subgradient Method, Proximal Gradient Method Yu-Xiang Wang CS292A (Based

Nonconvex Optimization for Communication Systems

A BFGS-SQP method for nonsmooth, nonconvex, …A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles Frank E

Nonconvex Nonsmooth Minimization via Cutting Planes and

Nonconvex Optimization Meets Low-Rank Matrix Factorization ...yc5/publications/NcxOverview_Arxiv.pdf · Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview Yuejie

Nonconvex Mixed-Integer Nonlinear Optimization