Nonlinear equations (cont.) and Optimization inOctave

Agnes Baran, Csaba Noszaly

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 1 / 31

Example

Division is sometimes a hard task, but in order to divide we do not actuallyneed to perform any division.For a given a ∈ R compute (approximate) 1

a . Define a function, which is

zero at 1a :

f (x) = a− 1

xand f ′(x) =

1

x2

Set up the Newton-iteration:

xk+1 = xk(2− a · xk)

And voila:

>> iter = @(x,a) x*(2-a*x) ;>> a=3;x=0.3;

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 2 / 31

>> x=iter(x,a)x = 0.330000000000000>> x=iter(x,a)x = 0.333300000000000>> x=iter(x,a)x = 0.333333330000000>> x=iter(x,a)x = 0.333333333333333

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 3 / 31

Example

Computing the normed version of a vector is a general task in numericalmath:

x = (x1, . . . , xn) → (x1, . . . , xn)

||x ||2It is somewhat surprising, that it can computed (approximately) withoutany division and ”squarerooting”. Let us examine the equation:

f (x) = a− 1

x2

It is clear, that f ( 1√a

) = 0, so we can apply the Newton method.

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 4 / 31

ExerciseDevelop the corresponding iterating rule.

ExerciseCompute the normed version of the vectors.

(1, 1), (−3, 2), (−6,−1)

Compare the results with the output of the builtin functions.

ExerciseCompute the normed version of the vectors, but this time use the 1-norm!

(1, 1), (−3, 2), (−6,−1)

Compare the results with the output of the builtin functions.

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 5 / 31

Optimization intro

For a given f : [a, b]→ R we search for a point ξ ∈ (a, b) for which

f (ξ) ≤ f (x)

if |x − ξ| is small enough. We say that f (ξ) is a local minimum and ξ is alocal minimizer. We know from calculus that

f ′(ξ) = 0

at a local minimizer.Example

Find a local minimizer for

f (x) = x5 − cos(x)− ex

on [0, 1] with the Newton method. First, we examine the plot of it:

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 6 / 31

-2.7

-2.6

-2.5

-2.4

-2.3

-2.2

-2.1

-2

0 0.2 0.4 0.6 0.8 1

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 7 / 31

Now it is clear that our function has a local minimizer on [0, 1]. Aftercomputing the derivatives:

f ′(x) = 5x4 + sin(x)− ex and f ′′(x) = 20x3 + cos(x)− ex

we can easily approximate the root of f ′, which is a minimizer of f withOctave:

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 8 / 31

>> f=@(x) x.ˆ5-cos(x)-exp(x);>> df=@(x) 5*x.ˆ4+sin(x)-exp(x);>> ddf=@(x) 20*x.ˆ3+cos(x)-exp(x);>> x=0.7x = 0.700000000000000>> x=x-df(x)/ddf(x)x = 0.730125169245811>> x=x-df(x)/ddf(x)x = 0.728187065551714>> x=x-df(x)/ddf(x)x = 0.728178496917075>> x=x-df(x)/ddf(x)x = 0.728178496750211>> x=x-df(x)/ddf(x)x = 0.728178496750211

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 9 / 31

In Octave, we have two possibilities for computing the zeros:The function fzero can be used for univariate functions. By default, itneeds an initial bracketing (an interval that consists the zero), but can beused also without specifying it:

>> fzero(df, [0, 1])ans = 0.728178496750212>> fzero(df, 0.6)ans = 0.728178496750211

The function fsolve can be applied for multivariate functions. Here weonly have to specify an initial guess of the zero:

>> fsolve(df, 0.6)ans = 0.728178501057600

As we see, the results can be differ slightly. Their behaviour can be finetuned, through setting appropriate parameters, for details see the help.

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 10 / 31

Example

Of all right angled triangle with a given hypotenuse,find that whose area isthe greatest. Denote the length of the hypotenuse by c and by x one ofthe other sides. Applying the Pythagorean theorem we get

√c2 − x2 for

the third side, therefore the area of our triangle is

A(x) = 0.5x√

c2 − x2

where 0 < x < c. Now, in order to avoid the difficulties of computing thederivatives, we omit the factor 0.5 and take square of the remainingexpression:

(2 ∗ A(x))2 = B(x) = x2(c2 − x2) = c2x2 − x4

Let us examine the plot of B with c = 1:

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 11 / 31

0

0.05

0.1

0.15

0.2

0.25

0 0.2 0.4 0.6 0.8 1

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 12 / 31

kett

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 13 / 31

Apply again the Newton method and compare the result with the outputof fzero :

>> B=@(x,c) cˆ2*x.ˆ2-x.ˆ4 ;>> dB=@(x,c) 2*cˆ2*x-4*x.ˆ3 ;>> ddB=@(x,c) 2*cˆ2-12*x.ˆ2 ;>> c=1;>> x=0.7x = 0.700000000000000>> x=x-dB(x,1)/ddB(x,1)x = 0.707216494845361>> x=x-dB(x,1)/ddB(x,1)x = 0.707106806711824>> x=x-dB(x,1)/ddB(x,1)x = 0.707106781186549>> x=x-dB(x,1)/ddB(x,1)x = 0.707106781186548>> fzero(@(x)dB(x,1),0.7)

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 13 / 31

fsolve for multivariate vector-function

Example

Solve the system below for x1, x2.

−4x1 + cos(2x1 − x2) = 3

−3x2 + sin x1 = 2 x1, x2 ∈ [−π, π]

Solution:

>> f=@(x) [ -4*x(1)+cos(2*x(1)-x(2))-3; -3*x(2)+sin(x(1))-2 ];>> x0=[0;0];>> [z,fz]=fsolve(f,x0)z =

-0.504059590609616-0.827661401511504

fz =-3.96460109186592e-093.15550252594221e-10

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 14 / 31

fsolve for multivariate real-function

Example

Solve the system below for x1, x2.

−4x1 ∗ (2− x2) + cos(2x1 − x2) = 3

First we plot the function:

>> xx=linspace(-pi,pi);>> [X,Y]=meshgrid(xx,xx);>> Z=X.*Y+cos(2*X-Y)+1;>> plot3(X,Y,Z)

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 15 / 31

-4-3-2-101234-4 -3 -2 -1 0 1 2 3 4

-10

-5

0

5

10

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 16 / 31

fsolve for multivariate real-function

Explanation:https://octave.sourceforge.io/octave/function/meshgrid.html

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 17 / 31

optimization with built-in functions

There is a simpler way to find a minimizer for a univariate function, wecan apply the function fminbnd . We pass a function and an interval, andfminbnd tries to find the minimal value on the given interval:

>> B=@(x) -x.ˆ2 + x.ˆ4 ;>> [xopt, fopt]=fminbnd(B,0.1,1)xopt = 0.707106781164670fopt = -0.250000000000000

Note that we changed the sign of the function B in order to getmaximization problem.

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 18 / 31

ExerciseFind a local minimizer for

f (x) = x5 − cos(x)− ex

using fminbnd .

ExerciseFind a local minimizer for

f (x) = xx − cos(x)− 0.1 · ex

using fminbnd .

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 19 / 31

optimization of multivariate functions

We look for the local minimizers of the function:

f (x1, x2) = x31 + x32 − 3x1 − 3x2

on [−2, 2]× [−2, 2]To get an overview of a function, we create its plot:

>> xx=linspace(-2,2);>> [X,Y]=meshgrid(xx,xx);>> Z=X.ˆ3+Y.ˆ3-3*X-3*Y;>> plot3(X,Y,Z)

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 20 / 31

-2-1.5-1

-0.500.511.5

2

-2-1.5-1-0.500.511.52-4-3-2-101234

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 21 / 31

The optimization methods often need an initial guess to start the process.You can guess good starting points, vectors by examining the contour plot.https://en.wikipedia.org/wiki/Contour_line

xx=linspace(-2,2);yy=xx;[X,Y]=meshgrid(xx,yy);Z=X.ˆ3+Y.ˆ3-3*X-3*Y;figure; contour(X,Y,Z)axis equal

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 22 / 31

-1.5

-1

-0.5

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 23 / 31

Another useful plot for R2 → R functions is a gradient plot.https://en.wikipedia.org/wiki/GradientIts most important property: the gradient at (x1, x2) has the direction ofgreatest increase of the function at (x1, x2).The corresponding Octave code:

xx=linspace(-2,2,11); yy=xx;[X,Y]=meshgrid(xx,yy);dX=3*X.ˆ2-3;dY=3*Y.ˆ2-3;hold on; quiver(X,Y,dX,dY)

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 24 / 31

-1

-0.5

0

0.5

1

1.5

2

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 25 / 31

optimization with built-in functions

The function fminsearch uses derivative-free method to approximate thelocal minimizer. The parameters: the function handle and the initial guess:

>> f=@(x) x(1)ˆ3+x(2)ˆ3-3*x(1)-3*x(2);>> [xo,fxo]=fminsearch(f,[0.5,0.5])xo =

0.999961405649954 1.000002133792563

fxo = -3.99999999551783

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 26 / 31

optimization with built-in functions

The function fminunc uses derivative-based method to approximate thelocal minimizer. The parameters: the function handle and the initial guess:

>> f=@(x) x(1)ˆ3+x(2)ˆ3-3*x(1)-3*x(2);>> [xo,fxo]=fminunc(f,[0.5,0.5])xo =

1.00000000468604 1.00000000468604

fxo = -4.00000000000000

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 27 / 31

ExercisePlot the function, its contour-lines and the gradient field on the givendomain. Approximate a local minimizer-maximizer!

f (x1, x2) =x316− x1 +

x1x22

4if (x1, x2) ∈ [−2.6]× [−2.6]

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 28 / 31

ExercisePlot the function, its contour-lines and the gradient field on the givendomain. Approximate a local minimizer-maximizer!

f (x1, x2) = sin(x1) cos(x2) if (x1, x2) ∈ [0, π]× [0, π]

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 29 / 31

ExercisePlot the Rosenbrock-function, its contour-lines and the gradient field.Approximate a local minimizer-maximizer!https://en.wikipedia.org/wiki/Rosenbrock_function

Agnes Baran, Csaba Noszaly Nonlinear equations (cont.) and Optimization in Octave 30 / 31