Root Finding without Derivativeshomepages.math.uic.edu/~jan/mcs471/rootfindsecant.pdf · Root...

Root Finding without Derivatives

1 Secant Methodderivation of the methoda Julia functionsuperlinear convergence

2 Method of False Positiona bad case for the secant methodcombine secant with bisection

MCS 471 Lecture 3(b)Numerical Analysis

Jan Verschelde, 22 June 2018

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 1 / 27

Root Finding without Derivatives

1 Secant Methodderivation of the methoda Julia functionsuperlinear convergence

2 Method of False Positiona bad case for the secant methodcombine secant with bisection

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 2 / 27

the secant line

Consider the secant line through (a, f (a)) and (b, f (b)):

y − f (b) =

(f (b)− f (a)

b − a

)(x − b).

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 3 / 27

derivation of the methodConsider the secant line through (a, f (a)) and (b, f (b)):

y − f (b) =

(f (b)− f (a)

b − a

)(x − b)

Set x = c and y = 0 and solve for c:

0− f (b) =

(f (b)− f (a)

b − a

)(c − b)

−f (b)

(b − a

f (b)− f (a)

)= c − b

Then we obtain the formula

c = b − f (b)

(b − a

f (b)− f (a)

)

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 4 / 27

the secant method

Rename a = x1, b = x2, c = x3:

x3 = x2−f (x2)

(x2 − x1

f (x2)− f (x1)

)︸ ︷︷ ︸

∆x

Input: f , x1, x2, N, δ, ε.

for k = 1,2, . . . ,N do

∆x := f (x2)

(x2 − x1

f (x2)− f (x1)

)x1 := x2x2 := x2 −∆xif |∆x | ≤ δ or |f (x2)| ≤ ε

report success, return x2

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 5 / 27

Root Finding without Derivatives

1 Secant Methodderivation of the methoda Julia functionsuperlinear convergence

2 Method of False Positiona bad case for the secant methodcombine secant with bisection

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 6 / 27

a Julia function

function secant(f::Function,x1::Float64,x2::Float64,dxtol::Float64=1.0e-8,fxtol::Float64=1.0e-8,N::Int64=10)

## Applies the secant method to the function f,# starting at the points x1 and x2.# Stops when the sequence reaches the point x# where |f(x)| < fxtol or when the distance between# two consecutive points is less than dxtol.# Failure is reported when the accuracy requirement# is not satisfied in N steps,# otherwise fail is false on return.#

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 7 / 27

the definition of the function

println("running the secant method...")fx1 = f(x1)fx2 = f(x2)for i = 1:N

dx = fx2*(x2 - x1)/(fx2 - fx1)x1 = x2 # store x2 for next stepx2 = x2 - dxfx1 = fx2fx2 = f(x2)

Observe that there is only one new function evaluationin each iteration of the loop: fx2 = f(x2).

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 8 / 27

printing the results and the stop criterion

stri = @sprintf("%3d", i)strx = @sprintf("%.16e", x2)strdx = @sprintf("%.4e", abs(dx))strfx = @sprintf("%.4e", abs(fx2))println("$stri : $strx $strdx $strfx")if((abs(fx2) < dxtol) | (abs(dx) < fxtol))

stri = string(i)println("succeeded after $stri steps")return [x2, false]

endendstrN = string(N)println("failed requirements after $strN steps")return [x2, true]

end

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 9 / 27

running a test

result = secant(cos,pi/4,2*pi/3)

is executed in the script:

$ julia secantmethod.jlrunning the secant method...step : root |dx| |f(x)|1 : 1.5521908171562901e+00 5.4220e-01 1.8604e-022 : 1.5716418753086792e+00 1.9451e-02 8.4555e-043 : 1.5707962802269275e+00 8.4560e-04 4.6568e-084 : 1.5707963267949021e+00 4.6568e-08 5.4899e-15

succeeded after 4 steps$

Observe the quadratic convergence.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 10 / 27

two exercises

Exercise 1: Consider running the secant method on x3 + x − 1and choose 0.5 as the first value for x1.Experiment with different values for the second point x2,choosing 0.500001, 0.6, and 0.7 for x2.For which choice of x2 does the secant method work best?

Exercise 2: The polynomial x3 + x − 1 has two complex conjugatedroots but the secant method cannot compute the complex roots,unless the arithmetic is complex.Adjust the Julia function so it works with initial values with nonzeroimaginary parts.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 11 / 27

Root Finding without Derivatives

1 Secant Methodderivation of the methoda Julia functionsuperlinear convergence

2 Method of False Positiona bad case for the secant methodcombine secant with bisection

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 12 / 27

superlinear convergence

TheoremAssume the secant method applied to f (x) = 0 converges:

1 xk → x∞, f (x∞) = 02 f ′(x∞) 6= 0.

Denote ek = xk − x∞, the error in the k-th step. Then,

ek+1 ≈ Ce(1+√

5)/2k ≈ Ce1.618

k ,

for some constant C.

Note that1 +√

52

is the golden ratio.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 13 / 27

a first lemma

LemmaUnder the assumptions and notations of the theorem:

ek+1 ≈12

f ′′(x∞)

f ′(x∞)ek−1ek .

Proof. Using the definition of xk+1, we find

ek+1 = xk+1 − x∞ = xk − f (xk )xk − xk−1

f (xk )− f (xk−1)− x∞.

We can replace xx+1 by xk + ek and xk by xk−1 + ek−1, so that

ek+1 = x∞ + ek − f (x∞ + ek )x∞ + ek − x∞ − ek−1

f (x∞ + ek )− f (x∞ + ek−1)− x∞. (1)

To simplify this expression, we apply the Taylor expansion off (x∞ + ek ) and f (x∞ + ek−1) about x∞.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 14 / 27

proof of the first lemma continued 1

f (x∞ + ek ) = f (x∞) + f ′(x∞)ek +12

f ′′(x∞)e2k + O(e3

k ),

f (x∞ + ek−1) = f (x∞) + f ′(x∞)ek−1 +12

f ′′(x∞)e2k−1 + O(e3

k−1).

Subtracting f (x∞ + ek−1) from f (x∞ + ek ):

f (x∞ + ek )− f (x∞ + ek−1)

= f ′(x∞)(ek − ek−1) + 12 f ′′(x∞)(e2

k − e2k−1) + O(e3

k )−O(e3k−1).

Since O(e3k )−O(e3

k−1) is of a smaller order than ek and ek−1 we omitthis term. Using e2

k − e2k−1 = (ek − ek−1)(ek + ek−1), we organize the

above expression as

f (x∞+ek )−f (x∞+ek−1) ≈ (ek−ek−1)(f ′(x∞)+f ′′(x∞)(ek +ek−1)). (2)

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 15 / 27

proof of the first lemma continued 2

The left of (2):

f (x∞ + ek )− f (x∞ + ek−1) ≈ (ek − ek−1)(f ′(x∞) + f ′′(x∞)(ek + ek−1)).

appears at the right of (1):

ek+1 = x∞ + ek − f (x∞ + ek )x∞ + ek − x∞ − ek−1

f (x∞ + ek )− f (x∞ + ek−1)− x∞.

so we derive the following expression

ek+1 ≈ ek − f (x∞ + ek )ek − ek−1

(ek − ek−1)(f ′(x∞) + f ′′(x∞)(ek + ek−1)).

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 16 / 27

proof of the first lemma continued 3

Using a Taylor expansion for f (x∞ + ek ) about x∞ (recall f (x∞) = 0)we have

ek+1 ≈ ek − ekf ′(x∞) + 1

2 f ′′(x∞)ek

f ′(x∞) + 12 f ′′(x∞)(ek + ek−1)

.

Now we put everything on the same denominator:

ek+1 ≈ ekf ′(x∞) + 1

2 f ′′(x∞)(ek + ek−1)− f ′(x∞)− 12 f ′′(x∞)ek

f ′(x∞) + 12 f ′′(x∞)(ek + ek−1)

,

which can be simplified as

ek+1 ≈ ek

12 f ′′(x∞)ek−1

f ′(x∞) + 12 f ′′(x∞)(ek + ek−1)

.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 17 / 27

end of the proof of the first lemma

Because ek → 0 as k →∞, 12 f ′′(x∞)(ek + ek−1) is negligible

compared to f ′(x∞), so we omit the second term in the denominator,to find the estimate

ek+1 ≈12

f ′′(x∞)

f ′(x∞)ekek−1.

Q.E.D.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 18 / 27

a second lemma

LemmaThere exists a positive real number r such that:

ek+1 ≈ Cek−1ek ⇒ e1+1/rk ≈ Ker

k , for some constants C and K .

Proof. Assuming the convergence rate is r , there exists someconstant A, so we can write

ek+1 ≈ Aerk and ek ≈ Aer

k−1 or(

1A

ek

)1/r

≈ ek−1.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 19 / 27

proof of the second lemma continued

Now we can replace the expressions for ek and ek−1 in the left handside of the equation of the second lemma:

ek+1 ≈ C(

1A

)1/r

e1/rk ek ≈ Be1+1/r

k .

Together with the assumption that ek+1 ≈ Aerk , we obtain

e1+1/rk ≈ A

B erk . So, we set K = A

B and the lemma is proven. Q.E.D.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 20 / 27

a third lemma

LemmaFor the r of the second lemma, we have

e1+1/rk ≈ Cer

k ⇒ r =1 +√

52

.

Proof. r satisfies the following equation

1 +1r

= r ⇒ r + 1 = r2 ⇒ r2 − r − 1 = 0.

The roots of r2 − r − 1 = 0 are r = 1±√

52 .

We take the positive value for r . Q.E.D.The constant r = 1+

√5

2 ≈ 1.618 is the golden ratio.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 21 / 27

Root Finding without Derivatives

1 Secant Methodderivation of the methoda Julia functionsuperlinear convergence

2 Method of False Positiona bad case for the secant methodcombine secant with bisection

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 22 / 27

a bad case for the secant method

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 23 / 27

Root Finding without Derivatives

1 Secant Methodderivation of the methoda Julia functionsuperlinear convergence

2 Method of False Positiona bad case for the secant methodcombine secant with bisection

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 24 / 27

combine secant with bisection

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 25 / 27

the method of false position or regula falsi

Input: [a,b] such that f (a)f (b) < 0;N is the maximum number of iterations;ε > 0, tolerance on |f (x)|.

for i = 1,2, . . . ,N do

c :=b f (a)− a f (b)

f (a)− f (b)

if |f (c)| ≤ ε then return c

if f (a)f (c) < 0 thenb := c

elsea := c.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 26 / 27

write a Julia function

Exercise 3: Write a Julia script to run the method of false position.Test your script by running the experiment of Exercise 1.

Numerical Analysis (MCS 471) Bisection and Fixed-Point Iterations L-3(b) 22 June 2018 27 / 27