Part 4 Nonlinear Programming

4.3 Successive Linear Programming

Basic Concept

0 0 0 0;f f f f

y y y y y y y

Approach 1: Direct Use of Linear Programs

The simplest and most direct use of the linearization construction is to replace the general nonlinear problem with a complete linearization of all problem functions at some selected estimate solution.

The linearized problem takes the form of a linear program and can be solved as such.

Case 1.1 The linearly constrained case

min

. .

Because ( ) is nonlinear, the optimal solution no

longer needs

f

s t

f

y

Ay b

y 0

y

to be confined to corner points of the

feasible region but can lie anywhere within the region.

Case 1.1The approximate problem

0min ;

. .

f

s t

y y

Ay b

y 0

*

*

0 0 * 0

0 0 0 * 0

0 * 0

0

How is , the solution of the approximate problem,

related to , the solution of the original problem?

Note,

; ;

or

0

Since is in the direction of steepest

f f

f f f

f

f

y

y

y y y y

y y y y y

y y y

y

* 0

ascent, the

vector is in a descent direction on the surface

of .f

y y

y

Bounded Line Search

*

0

0

Since is a corner point of the feasible region,

and since is feasible, all points on the line between

the two will be feasible.

The line search can be confined to the bounded line

segment,

y

y

y y

* 0 where 0 1 y y

Frank-Wolfe Algorithm

( )

( )

( )

Step 1: Calculate

Step 2: Solve the LP subproblem

min

. .

Let be the optimal solution.

k

k

k

f

f

s t

y

y z

Az b

z 0

z

( )

( ) ( ) ( )

0 1

( 1) ( ) ( ) ( ) ( )

( 1)

( 1) ( )

( 1)

( 1) ( )

( 1)

Step 3: Find which solves

min

Step 4: Calculate

Step 5: Convergence check.

Otherwise, go t( ) ( )

( )

k

k k k

k k k k k

k

k k

k

k k

k

f

f

f f

f

y z y

y y z y

y

y y

y

y y

y

o step 1.

Remark

( ) ( ) ( )

( ) ( ) ( )

( )

( ) ( ) ( )

If the objective function is convex, then

for all feasible points. This implies

min min ; ; ...lower bound

It is also clear that

min

Thus, ; g

k k k

k k k

k

k k k

f f f

f f f

f f

f f

y y y y y

y y y z y

y y

y z y

ives an estimate of the

improvement in objective function.

Case 1.2The general LP case

min

. .

0 1, 2, ,

0 1, 2, ,

1, 2, ,

j

k

i i i

f

s t

g j J

h k K

U x L i N

y

y

y

Direct Linear Approximation

( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( )

( )

min

. .

0 1, 2, ,

0 1, 2, ,

1, 2, ,

Note that even if is a feasible point of the original

nonlinear proble

t t t

t t tj j

t t tk k

i i i

t

f f

s t

g g j J

h h k K

U x L i N

y y y y

y y y y

y y y y

y

( 1)m, there is no assurance that will

be feasible!

ty

Remark

In order attain convergence to the true optimum, it is sufficient that at each iteration an improvement be made in both the objective function and constraint infeasibility.

This type of monotonic behavior will occur if the problem functions are mildly nonlinear.

Approach 2Separable Programming

The motivation for this technique stems from the observation that a good way of improving the linear approximation over a large interval is to partition the interval into subintervals and construct individual linear approximation over each subinterval, i.e., piecewise linear approximation.

Case 2.1Single-Variable Functions

Line Segment in Interval k

( 1) ( )

( ) ( )( 1) ( )

( ) ( 1)

The equation of line in interval is

k kk k

k k

k k

k

f ff x f x x

x x

x x x

Line Segment in Interval k

( ) ( ) ( 1) ( 1)

( ) ( 1) ( ) ( 1)

( 1) ( )( ) ( ) ( ) ( 1) ( 1) ( )

( 1) ( )

( 1) ( )( ) ( 1) ( 1) ( 1) ( )

( 1) ( )

( ) ( 1) ( 1) ( )

(

where

1 and , 0

( )

k k k k

k k k k

k kk k k k k k

k k

k kk k k k k

k k

k k k k

k

x x x

f ff x f x x x

x x

f ff x x

x x

f f f

) ( ) ( 1) ( 1)k k kf f

General Formula

(1) ( )

( ) ( )

1

( ) ( )

1

( ) ( )

1

( ) ( )

For any point in

where

1 and 0 1, 2, ,

0 if 1 and 1, 2, , 2

K

Kk k

k

Kk k

k

Kk k

k

i j

x x x

x x

f x f

k K

j i i K

Case 2.2Multivariable Separable Functions

1

A function is said to be separable if it can be

expressed as the sum of the single-variable functions

that each involve only one of the N variables, i.e.,

N

i ii

f f x

x

General Formula

1 2

(1) (2) ( )

1 2

( ) ( ) ( ) ( ) ( ) ( )1 1 2 2

1 1 1

( ) ( )

1 1

Let

where 1, 2, ,

Then

, , ,

N

i

Ki i i i i

N

KK Kk k k k k k

N Nk k k

KNk ki i

i k

L x x x U

i N

f x x x

f f f

f

General Formula

( ) ( ) ( ) ( )

1

( )

1

( )

( ) ( )

In the previous slide,

and

1

0 1, 2, ,

0 if 1 and 1,2, , 2

and

1,2, ,

i

i

Kk k k k

i i i i i ik

Kki

k

ki i

m ni i i

x x f f x

k K

n m m K

i N

Restricted Basis Entry

Prior to entering one lambda into the basis (which will make it nonzero), a check should be made to ensure that no more than one other lambda associated with the same x_i is in the basis.

If there is one such lambda in the basis, it has to be adjacent.

Example

41 2 1 2

21 2 1 2

1

2

max ,

. .

, 9 2 3 0

0

0

f x x x x

s t

g x x x x

x

x

1 2 1 1 2 2

41 1 1

2 2 2

1 2 1 1 2 2

21 1 1

2 2 2

,

,

2

9 3

f x x f x f x

f x x

f x x

g x x g x g x

g x x

g x x

1Construct the approximation over 0 3

and use 4 equidistant points

x

k

1 0 0 0

2 1 1 -2

3 2 16 -8

4 3 81 -18

( )1kx ( )

1kf ( )

1kg

(1) (2) (3) (4)1 1 1 1 1 1

(1) (2) (3) (4)1 1 1 1 1 1

(1) (2) (3) (4)1 1 1 1

Thus,

0 1 16 81

0 2 8 18

1

f x

g x

1 2 1 1 2 2

1 2 1 1 2 2

(2) (3) (4)1 1 1 2

(2) (3) (4)1 1 1 2 3

(1) (2) (3) (4)1 1 1 1

(1) (2) (3) (4)1 1 1 1 2 3

max ,

. .

,

2 8 18 3 9 0

( 2 8 18 3 9)

1

, , , , , 0

f x x f x f x

s t

g x x g x g x

x

x x

x x

Homework