sceintific computing

8/12/2019 sceintific computing

1/55

Nonlinear Equations

Numerical Methods in One Dimension

Methods for Systems of Nonlinear Equations

Scientific Computing: An Introductory SurveyChapter 5 Nonlinear Equations

Prof. Michael T. Heath

Department of Computer ScienceUniversity of Illinois at Urbana-Champaign

Copyright c 2002. Reproduction permittedfor noncommercial, educational use only.

Michael T. Heath Scientific Computing 1 / 55


2/55

Nonlinear Equations



Outline

1 Nonlinear Equations

2 Numerical Methods in One Dimension

3




3/55

Nonlinear Equations



Nonlinear Equations

Solutions and Sensitivity

Convergence

Nonlinear Equations

Given functionf, we seek valuexfor which

f(x) = 0

Solutionxisrootof equation, orzeroof functionf

So problem is known as root finding orzero finding



4/55

Nonlinear Equations



Nonlinear Equations


Convergence

Nonlinear Equations

Two important cases

Single nonlinear equation in one unknown, where

f:R

R

Solution is scalarxfor whichf(x) = 0

System ofncouplednonlinear equations inn unknowns,

wheref: Rn Rn

Solution is vector xfor which all components of fare zero

simultaneously,f(x) = 0



5/55

Nonlinear Equations



Nonlinear Equations


Convergence

Examples: Nonlinear Equations

Example of nonlinear equation in one dimension

x2 4sin(x) = 0

for whichx = 1.9is one approximate solution

Example of system of nonlinear equations in two

dimensions

x21 x2+ 0.25 = 0

x1+x22+ 0.25 = 0

for whichx=

0.5 0.5T

is solution vector


N li E i N li E i


6/55

Nonlinear Equations



Nonlinear Equations


Convergence

Existence and Uniqueness

Existence and uniqueness of solutions are more

complicated for nonlinear equations than for linear

equations

For functionf: R R,bracket is interval[a, b]for whichsign offdiffers at endpoints

Iff is continuous andsign(f(a))= sign(f(b)), thenIntermediate Value Theorem implies there isx

[a, b]

such thatf(x) = 0

There is no simple analog for n dimensions


N li E ti N li E ti


7/55

Nonlinear Equations



Nonlinear Equations


Convergence

Examples: One Dimension

Nonlinear equations can have any number of solutions

exp(x) + 1 = 0has no solution

exp(x) x= 0has one solutionx2 4 sin(x) = 0has two solutionsx3 + 6x2 + 11x

6 = 0has three solutions

sin(x) = 0has infinitely many solutions


Nonlinear Equations Nonlinear Equations


8/55

Nonlinear Equations



Nonlinear Equations


Convergence

Example: Systems in Two Dimensionsx21

x2+ = 0

x1+x22+ = 0




9/55

Nonlinear Equations



Nonlinear Equations


Convergence

Multiplicity

Iff(x) =f(x) =f(x) = =f(m1)(x) = 0butf(m)(x)= 0(i.e.,mth derivative is lowest derivative offthat does not vanish atx), then rootx hasmultiplicity m

Ifm = 1(f(x) = 0andf(x)= 0), thenx issimplerootMichael T. Heath Scientific Computing 9 / 55



10/55

Nonlinear Equations



Nonlinear Equations


Convergence

Sensitivity and Conditioning

Conditioning of root finding problem is opposite to that forevaluating function

Absolute condition number of root finding problem for root

x off: R

R is1/

|f(x)

|Root is ill-conditioned if tangent line is nearly horizontal

In particular, multiple root (m >1) is ill-conditioned

Absolute condition number of root finding problem for root

x off: Rn Rn isJ1f (x), whereJf isJacobianmatrix off,

{Jf(x)}ij =fi(x)/xjRoot is ill-conditioned if Jacobian matrix is nearly singular




11/55

Nonlinear Equations



Nonlinear Equations


Convergence





12/55

Nonlinear Equations



Nonlinear Equations


Convergence


What do we mean by approximate solution xto nonlinearsystem,

f

(x

) 0 or

x

x

0 ?First corresponds to small residual, second measures

closeness to (usually unknown) true solutionx

Solution criteria are not necessarily small simultaneously

Small residual implies accurate solution only if problem is

well-conditioned




13/55

q



q


Convergence

Convergence Rate

For general iterative methods, define error at iteration k by

ek =xk x

wherexk is approximate solution and x is true solution

For methods that maintain interval known to containsolution, rather than specific approximate value for

solution, take error to be length of interval containing

solution

Sequence converges with rater if

limk

ek+1ekr =C

for some finite nonzero constant C




14/55

q



q


Convergence

Convergence Rate, continued

Some particular cases of interest

r= 1: linear (C 1: superlinear

r= 2: quadratic

Convergence Digits gained

rate per iteration

linear constant

superlinear increasing

quadratic double


Nonlinear Equations Bisection Method


15/55



Fixed-Point Iteration and Newtons Method

Additional Methods

Interval Bisection Method

Bisectionmethod begins with initial bracket and repeatedlyhalves its length until solution has been isolated as accurately

as desired

while((b a)> tol) dom=a+ (b

a)/2

ifsign(f(a)) = sign(f(m)) thena= m

else

b=mend

end

< interactive example >


Nonlinear Equations Bisection Method
http://www.cse.uiuc.edu/iem/nonlinear_eqns/Bisection/http://www.cse.uiuc.edu/iem/nonlinear_eqns/Bisection/


16/55




Additional Methods

Example: Bisection Method

f(x) =x2

4sin(x) = 0

a f(a) b f(b)

1.000000 2.365884 3.000000 8.4355201.000000 2.365884 2.000000 0.3628101.500000

1.739980 2.000000 0.362810

1.750000 0.873444 2.000000 0.3628101.875000 0.300718 2.000000 0.3628101.875000 0.300718 1.937500 0.0198491.906250 0.143255 1.937500 0.0198491.921875

0.062406 1.937500 0.019849

1.929688 0.021454 1.937500 0.0198491.933594 0.000846 1.937500 0.0198491.933594 0.000846 1.935547 0.0094911.933594 0.000846 1.934570 0.0043201.933594 0.000846 1.934082 0.0017361.933594 0.000846 1.933838 0.000445Michael T. Heath Scientific Computing 16 / 55

Nonlinear Equations

N i l M h d i O Di i

Bisection Method

Fi d P i I i d N M h d


17/55




Additional Methods

Bisection Method, continued

Bisection method makes no use of magnitudes of function

values, only their signs

Bisection is certain to converge, but does so slowly

At each iteration, length of interval containing solution

reduced by half, convergence rate is linear, withr= 1and

C= 0.5

One bit of accuracy is gained in approximate solution for

each iteration of bisection

Given starting interval[a, b], length of interval afterk

iterations is(b a)/2k, so achieving error tolerance oftolrequires

log2

b a

tol

iterations, regardless of functionf involved


Nonlinear Equations

N i l M th d i O Di i

Bisection Method

Fi d P i t It ti d N t M th d


18/55




Additional Methods

Fixed-Point Problems

Fixed pointof given functiong : R R is valuex such thatx=g(x)

Many iterative methods for solving nonlinear equations use

fixed-point iterationscheme of form

xk+1=g(xk)

where fixed points forg are solutions forf(x) = 0

Also calledfunctional iteration, since functiong is appliedrepeatedly to initial starting value x0

For given equationf(x) = 0, there may be many equivalentfixed-point problemsx= g(x)with different choices forg


Nonlinear Equations


Bisection Method

Fixed Point Iteration and Newtons Method


19/55



Fixed-Point Iteration and Newton s Method

Additional Methods

Example: Fixed-Point Problems

Iff(x) =x2 x 2,then fixed points of each of functionsg(x) =x2 2

g(x) = x+ 2g(x) = 1 + 2/x

g(x) = x2 + 2

2x 1are solutions to equationf(x) = 0


Nonlinear Equations


Bisection Method

Fixed Point Iteration and Newtons Method


20/55




Additional Methods

Example: Fixed-Point Problems


Nonlinear EquationsNumerical Methods in One Dimension

Bisection MethodFixed-Point Iteration and Newtons Method


21/55




Additional Methods

Example: Fixed-Point Iteration





22/55



Fixed Point Iteration and Newton s Method

Additional Methods

Example: Fixed-Point Iteration





23/55



Fixed Point Iteration and Newton s Method

Additional Methods

Convergence of Fixed-Point Iteration

Ifx

=g(x

)and|g

(x

)|< 1, then there is intervalcontainingx such that iteration

xk+1=g(xk)

converges tox

if started within that intervalIf|g(x)|> 1, then iterative scheme divergesAsymptotic convergence rate of fixed-point iteration is

usually linear, with constantC=

|g(x)

|But ifg(x) = 0, then convergence rate is at leastquadratic




http://www.cse.uiuc.edu/iem/nonlinear_eqns/FixedPoint/http://www.cse.uiuc.edu/iem/nonlinear_eqns/FixedPoint/


24/55

Methods for Systems of Nonlinear Equations Additional Methods

Newtons Method

Truncated Taylor series

f(x+h)f(x) +f(x)h

is linear function ofh approximatingf nearx

Replace nonlinear functionfby this linear function, whosezero ish =f(x)/f(x)Zeros of original function and linear approximation are not

identical, so repeat process, giving Newtons method

xk+1=xk f(xk)f(xk)





25/55


Newtons Method, continued

Newtons method approximates nonlinear functionf nearxk bytangent lineatf(xk)





26/55


Example: Newtons Method

Use Newtons method to find root of

f(x) =x2 4 sin(x) = 0Derivative is

f(x) = 2x 4cos(x)so iteration scheme is

xk+1=xk x2k 4 sin(xk)

2xk 4cos(xk)Takingx0= 3as starting value, we obtain

x f(x) f(x) h

3.000000 8.435520 9.959970 0.8469422.153058 1.294772 6.505771 0.1990191.954039 0.108438 5.403795 0.0200671.933972 0.001152 5.288919 0.0002181.933754 0.000000 5.287670 0.000000





27/55


Convergence of Newtons Method

Newtons method transforms nonlinear equationf(x) = 0into fixed-point problemx = g(x), where

g(x) =x f(x)/f(x)and hence

g(x) =f(x)f(x)/(f(x))2

Ifx is simple root (i.e.,f(x) = 0andf(x)= 0), theng(x) = 0

Convergence rate of Newtons method for simple root is

thereforequadratic(r= 2)

But iterations must start close enough to root to converge




M h d f S f N li E i


Addi i l M h d
http://www.cse.uiuc.edu/iem/nonlinear_eqns/Newton/http://www.cse.uiuc.edu/iem/nonlinear_eqns/Newton/


28/55


Newtons Method, continued

For multiple root, convergence rate of Newtons method is only

linear, with constantC= 1 (1/m), wherem is multiplicity

k f(x) =x2

1 f(x) =x2

2x+ 1

0 2.0 2.01 1.25 1.52 1.025 1.253 1.0003 1.125

4 1.00000005 1.06255 1.0 1.03125



M th d f S t f N li E ti


Additi l M th d


29/55


Secant Method

For each iteration, Newtons method requires evaluation of

both function and its derivative, which may be inconvenient

or expensive

Insecant method, derivative is approximated by finitedifference using two successive iterates, so iteration

becomes

xk+1=xk

f(xk)

xk xk1f(xk) f(xk1)

Convergence rate of secant method is normally

superlinear, withr1.618





Additional Methods


30/55


Secant Method, continued

Secant method approximates nonlinear functionf

by secant

line through previous two iterates






Additional Methods
http://www.cse.uiuc.edu/iem/nonlinear_eqns/Secant/http://www.cse.uiuc.edu/iem/nonlinear_eqns/Secant/


31/55


Example: Secant Method

Use secant method to find root off(x) =x2 4 sin(x) = 0

Takingx0= 1andx1= 3as starting guesses, we obtain

x f(x) h

1.000000 2.3658843.000000 8.435520 1.5619301.438070 1.896774 0.2867351.724805 0.977706 0.3050292.029833 0.534305

0.107789

1.922044 0.061523 0.0111301.933174 0.003064 0.0005831.933757 0.000019 0.0000041.933754 0.000000 0.000000





Additional Methods


32/55


Higher-Degree Interpolation

Secant method uses linear interpolation to approximatefunction whose zero is sought

Higher convergence rate can be obtained by using

higher-degree polynomial interpolation

For example, quadratic interpolation (Mullers method) has

superlinear convergence rate withr1.839Unfortunately, using higher degree polynomial also has

disadvantagesinterpolating polynomial may not have real roots

roots may not be easy to compute

choice of root to use as next iterate may not be obvious





Additional Methods


33/55


Inverse Interpolation

Good alternative isinverse interpolation, wherexk areinterpolated as function ofyk =f(xk)by polynomialp(y),so next approximate solution isp(0)

Most commonly used for root finding is inverse quadratic

interpolation





Additional Methods


34/55


Inverse Quadratic Interpolation

Given approximate solution valuesa,b,c, with functionvaluesfa,fb,fc, next approximate solution found by fittingquadratic polynomial toa,b,c as function offa,fb,fc, thenevaluating polynomial at0

Based on nontrivial derivation using Lagrange

interpolation, we compute

u=fb/fc, v=fb/fa, w=fa/fc

p=v(w(u w)(c b) (1 u)(b a))q= (w

1)(u

1)(v

1)

then new approximate solution isb+p/q

Convergence rate is normallyr1.839< interactive example >





Additional Methods
http://www.cse.uiuc.edu/iem/nonlinear_eqns/InverseInterpolation/http://www.cse.uiuc.edu/iem/nonlinear_eqns/InverseInterpolation/


35/55

y q

Example: Inverse Quadratic Interpolation

Use inverse quadratic interpolation to find root of

f(x) =x2 4 sin(x) = 0

Takingx = 1,2, and3 as starting values, we obtain

x f(x) h1.000000 2.3658842.000000 0.3628103.000000 8.4355201.886318

0.244343

0.113682

1.939558 0.030786 0.0532401.933742 0.000060 0.0058151.933754 0.000000 0.0000111.933754 0.000000 0.000000





Additional Methods


36/55

y q

Linear Fractional Interpolation

Interpolation using rational fraction of form(x) =

x uvx w

is especially useful for finding zeros of functions having

horizontal or vertical asymptotes

has zero atx = u, vertical asymptote atx =w/v, andhorizontal asymptote aty= 1/v

Given approximate solution valuesa,b,c, with functionvaluesfa,fb,fc, next approximate solution isc+h, where

h= (a c)(b c)(fa fb)fc

(a c)(fc fb)fa (b c)(fc fa)fbConvergence rate is normallyr1.839, same as forquadratic interpolation (inverse or regular)





Additional Methods


37/55

Example: Linear Fractional Interpolation

Use linear fractional interpolation to find root off(x) =x2 4 sin(x) = 0

Takingx = 1,2, and3 as starting values, we obtain

x f(x) h

1.000000 2.3658842.000000 0.3628103.000000 8.4355201.906953 0.139647 1.0930471.933351 0.002131 0.0263981.933756 0.000013 0.0004061.933754 0.000000 0.000003






Additional Methods
http://www.cse.uiuc.edu/iem/nonlinear_eqns/LinearFractional/http://www.cse.uiuc.edu/iem/nonlinear_eqns/LinearFractional/


38/55

Safeguarded Methods

Rapidly convergent methods for solving nonlinear

equations may not converge unless started close to

solution, but safe methods are slow

Hybrid methods combine features of both types ofmethods to achieve both speed and reliability

Use rapidly convergent method, but maintain bracket

around solution

If next approximate solution given by fast method fallsoutside bracketing interval, perform one iteration of safe

method, such as bisection





Additional Methods


39/55

Safeguarded Methods, continued

Fast method can then be tried again on smaller interval

with greater chance of success

Ultimately, convergence rate of fast method should prevail

Hybrid approach seldom does worse than safe method,

and usually does much better

Popular combination is bisection and inverse quadratic

interpolation, for which no derivatives required





Additional Methods


40/55

Zeros of Polynomials

For polynomialp(x)of degreen, one may want to find allnof its zeros, which may be complex even if coefficients are

real

Several approaches are availableUse root-finding method such as Newtons or Mullersmethod to find one root, deflate it out, and repeat

Form companion matrix of polynomial and use eigenvalueroutine to compute all its eigenvalues

Use method designed specifically for finding all roots ofpolynomial, such as Jenkins-Traub




Fixed-Point IterationNewtons Method

Secant Updating Methods


41/55

Systems of Nonlinear Equations

Solving systems of nonlinear equations is much more difficult

than scalar case because

Wider variety of behavior is possible, so determining

existence and number of solutions or good starting guess

is much more complex

There is no simple way, in general, to guarantee

convergence to desired solution or to bracket solution to

produce absolutely safe method

Computational overhead increases rapidly with dimension

of problem







42/55

Fixed-Point Iteration

Fixed-point problemforg :R

n

Rn

is to find vectorxsuchthat

x= g(x)

Correspondingfixed-point iteration is

xk+1=g(xk)

If(G(x))< 1,where isspectral radius andG(x)isJacobian matrix ofg evaluated at x, then fixed-point

iteration converges if started close enough to solution

Convergence rate is normally linear, with constant Cgivenby spectral radius(G(x))

IfG(x) =O, then convergence rate is at least quadratic







43/55

Newtons Method

Inndimensions,Newtons methodhas formxk+1=xk J(xk)1f(xk)

whereJ(x)is Jacobian matrix of f,

{J(x)}ij = fi(x)xjIn practice, we do not explicitly invert J(xk), but insteadsolve linear system

J(xk)sk =f(xk)forNewton step sk, then take as next iterate

xk+1=xk+ sk







44/55

Example: Newtons Method

Use Newtons method to solve nonlinear system

f(x) =

x1+ 2x2 2x21+ 4x

22 4

= 0

Jacobian matrix isJf(x) = 1 2

2x1 8x2

If we takex0=

1 2T

, then

f(x0) = 313 , Jf(x0) =

1 22 16

Solving system

1 22 16

s0=

313

givess0=

1.830.58

,

so x1=x0+ s0= 0.83 1.42

T







45/55

Example, continued

Evaluating at new point,

f(x1) =

0

4.72

, Jf(x1) =

1 2

1.67 11.3

Solving system 1 2

1.67 11.3 s1=

0

4.72 gives

s1=

0.64 0.32T, so x2=x1+ s1= 0.19 1.10TEvaluating at new point,

f(x2) = 0

0.83 , Jf(x2) =

1 2

0.38 8.76Iterations eventually convergence to solutionx =

0 1

T< interactive example >






C f N M h d
http://www.cse.uiuc.edu/iem/nonlinear_eqns/Newton2D/http://www.cse.uiuc.edu/iem/nonlinear_eqns/Newton2D/


46/55

Convergence of Newtons Method

Differentiating corresponding fixed-point operatorg(x) =x J(x)1f(x)

and evaluating at solutionx gives

G(x) =I (J(x)1J(x) +n

i=1

fi(x)Hi(x

)) =O

whereHi(x)is component matrix of derivative of J(x)1

Convergence rate of Newtons method for nonlinearsystems is normallyquadratic, provided Jacobian matrix

J(x)is nonsingular

But it must be started close enough to solution to converge






C t f N t M th d


47/55

Cost of Newtons Method

Cost per iteration of Newtons method for dense problem in ndimensions is substantial

Computing Jacobian matrix costsn2 scalar functionevaluations

Solving linear system costsO(n3)operations






S t U d ti M th d


48/55


Secant updatingmethods reduce cost by

Using function values at successive iterates to buildapproximate Jacobian and avoiding explicit evaluation ofderivatives

Updating factorization of approximate Jacobian rather thanrefactoring it each iteration

Most secant updating methods have superlinear but not

quadratic convergence rate

Secant updating methods often cost less overall than

Newtons method because of lower cost per iteration






B d M th d


49/55

Broydens Method

Broydens method is typical secant updating method

Beginning with initial guessx0 for solution and initial

approximate JacobianB0, following steps are repeated

until convergence

x0=initial guessB0=initial Jacobian approximation

fork= 0, 1, 2, . . .SolveBk sk =f(xk)forskxk+1=xk+ skyk =f(xk+1) f(xk)Bk+1=Bk+ ((ykBksk)sTk)/(sTk sk)

end






Broydens Method continued


50/55

Broydens Method, continued

Motivation for formula forBk+1is to make least change to

Bk subject to satisfyingsecant equation

Bk+1(xk+1 xk) =f(xk+1) f(xk)

In practice, factorization ofBk is updated instead of

updatingBk directly, so total cost per iteration is only O(n2)






Example: Broydens Method


51/55

Example: Broyden s Method

Use Broydens method to solve nonlinear system

f(x) =

x1+ 2x2 2x21+ 4x

22 4

= 0

Ifx0= 1 2T

, thenf(x0) = 3 13T

, and we choose

B0=Jf(x0) =

1 22 16

Solving system

1 22 16

s0=

313

givess0=

1.830.58

, so x1=x0+ s0=

0.831.42



52/55





Example continued


53/55

Example, continued

Evaluating at new pointx2 givesf(x2) =

01.08

, so

y1=f(x2) f(x1) =

03.64

From updating formula, we obtain

B2=

1 2

0.34 15.3

+

0 0

1.46 0.73

=

1 2

1.12 14.5

Iterations continue until convergence to solution x =

01



Nonlinear Equations



Fixed-Point Iteration

Newtons Method


Robust Newton Like Methods
http://www.cse.uiuc.edu/iem/nonlinear_eqns/Broyden/http://www.cse.uiuc.edu/iem/nonlinear_eqns/Broyden/


54/55

Robust Newton-Like Methods

Newtons method and its variants may fail to converge

when started far from solution

Safeguards can enlarge region of convergence of

Newton-like methods

Simplest precaution isdamped Newton method, in whichnew iterate is

xk+1=xk+ksk

wheresk is Newton (or Newton-like) step and k is scalar

parameter chosen to ensure progress toward solution

Parameterk reduces Newton step when it is too large,butk = 1suffices near solution and still yields fastasymptotic convergence rate



55/55

Documents

sceintific computing