Solving Nonlinear Equation - CAIG Lab - National Chiao ...caig.cs.nctu.edu.tw/course/NM/chap1.pdf · Solving Nonlinear Equation x 0 x 2 x 1 f ... System of n coupled nonlinear equations

Solving Nonlinear Equation

0x1x2x

)( 1xf

)( 0xf

Root r

Numerical Methods © Wen-Chieh Lin 2

Nonlinear Equations

Given function f, we find value x for which

Solution x is root of equation, or zero offunction f

So problem is known as root finding or zerofinding

0)( xf


Nonlinear Equations: two cases Single nonlinear equation in one unknown, where

Solution is scalar x for which f(x) = 0

System of n coupled nonlinear equations in nunknowns, where

Solution is vector x for which all components of fare zero simultaneously, f(x) = 0


Example of 1-D nonlinear equation

for which x = 0.3604 is one approximate solution

Example of system of nonlinear equations in twodimensions

for which x = [-1.8, 0.8] is one approximatesolution vector

1

4

2

22

21

1

xe

xxx

Examples: Nonlinear Equations

0)sin(3 xexx


Multiplicity If f(R) = f’(R) = f”(R) = … = f(m-1)(R) = 0 but

f(m)(R) ≠ 0, then root R has multiplicity m

If m = 1 (f(R) = 0, f’(R) ≠ 0 ), then R is simpleroot


Bisection method begins with initial bracket andrepeatedly halves its length until solution hasbeen isolated as accurately as desired

http://www.cse.uiuc.edu/iem/nonlinear_eqns/Bisection/

Interval Halving (Bisection)

while |b-a| > tol,m = a + (b-1)/2;If f(a)*f(m) < 0,

b = m;else

a = m;end;

end;

a bm


Bisection (cont.) Simple and guaranteed to work if

f is continuous in [a, b] [a, b] brackets a root

Needed iterations to achieve a specified accuracy isknown in advance Error after n iterations < |b - a| / 2n

Slow to converge Good for initial guess for other root finding

algorithms Finding the initial bracket may be a problem if f is not

given explicitly


Use graphing to assist root finding

Set the initial bracketDetect multiple roots


Can we find a root in a better way?

Bisection only utilizes function values f(x)We can find a root with fewer iterations if

other information is used linear approximation of f(x)

Secant line secant methodTangent line Newton’s method

Polynomial approximation of f(x)Muller’s method


Secant Method Approximate a function by a straight line

Compute the intersection of the line and x-axis

)()()(

)()(

10

10

1

21

xfxfxx

xfxx

)()()(

)(10

10112 xfxf

xxxfxx

0x1x2x

)( 1xf

)( 0xf

Root r


Secant Method (cont.) Update endpoints

Repeat

)()()(

)(1

11

nn

nnnnn xfxf

xxxfxx

0x

1x

)( 1xf

)( 0xf Root r

withswap,)()(if 1010 xxxfxf


Example f(x) = 3x + sin(x) –exp(x) Find the root in [0, 1] http://www.cse.uiuc.edu/iem/nonlinear_eqns/Secant/

0 0.5 1 1.5 2-1

-0.5

0

0.5

1

1.5


Method of False Position

Problem of secant method

RemedyAlways bracket a root in the interval [x0, x1]How to do this?


Newton’s methodBetter approximation using the first derivative

)( 0xf

10 xx 0x1x

10

00

)()(')tan(

xxxf

xf

)(')(

0

001 xf

xfxx

)(')(

1n

nnn xf

xfxx


Interpretation of Newton’s method Truncated Taylor series

is a linear function of h approximating f near xReplace nonlinear function f by this function,

whose zero is h = - f(x)/f’(x) Zeros of original function and linear

approximation are not identical, so repeatprocess

)(')()( xhfxfhxf

)(')(

1n

nnn xf

xfxx


Example: Newton’s method

http://www.cse.uiuc.edu/iem/nonlinear_eqns/Newton/


Comparison of Secant andNewton’s methods

)()()(

)(1

11

nn

nnnnn xfxf

xxxfxx

)('

)(1

n

nnn xf

xfxx

Secant method Newton’s method


Pros and Cons of Newton’s method

Pros efficient

ConsNeed to know the derivative function


)(')(

1n

nnn xf

xfxx

When will Newton’s method not converge?

x1=x6, loop

passingmaximum/minimum


Muller’s method Instead of linear approximation, Muller’s

method uses quadratic approximate Evaluation of derivatives are not required See the textbook for details


Fixed-point Iteration Method Rearrange f(x)=0 into an equivalent form x=g(x)

f(x) = x –g(x) = 0 If r is a root of f, then f(r) = r-g(r) = 0

r=g(r) In iterative form,

Also called function iteration For given equation f(x)=0, there may be many

equivalent fixed-point problems x=g(x) with differentchoice of g

Will the method always converge?

)(1 nn xgx


Example: Fixed-point Iteration

032)( 2 xxxf

32)(1 xxg2

3)(2

xxg


Example: Fixed-point Iteration (cont.)

032)( 2 xxxf

23

)(2

3

x

xg

Diverge!


Convergence Rate

For general iterative methods, define error atiteration n by

en = xn –Rwhere xn is approximate solution and R is truesolution

For methods that maintain interval known tocontain solution, rather than specificapproximate value for solution, take error to belength of interval containing solution


Convergence Rate (cont.) Sequence converges with rate r if

for some finite nonzero constant C Some cases of interest

r=1: linear (C<1) r>1: superlinear r=2: quadratic

Ce

er

n

n

n

1lim


Convergence Rate of Bisection

Length of interval containing solution reducedby half at each iteration

Linearly convergent r = 1C = 0.5


Convergence of Fixed-point Iteration

If R = g(R) and |g’(R)| < 1, then there is aninterval containing R such that iteration

xn+1 = g(xn)converges to R if started within that interval

If |g’(R)| > 1, then iterative scheme diverges


Proof of Convergent Condition

nnn ege )('1

)()()(1 nnn xgRgxgRxR

nn

nn xR

xRxgRg

xR

)()(

1

))(('1 nnn xRgxR Mean Value Theorem

),( nn xR

Ce

er

n

n

n

1limFixed-point iteration is linearly convergent

)(' RgC


Convergence of Newton’s MethodRepresent Newton’s method in fixed-point

iteration form

Condition for convergence

)()(')(

1 nn

nnn xg

xfxf

xx

1)]('[

)(")()(' 2

xfxfxf

xg


Convergence rate of Newton’s method)()(1 nn xgRgxR

2)()(")()(')()( 2nnn xRgxRRgRgxg

0)(')]('[

)(")()(' 0)(

2 Rgxf

xfxfxg Rf

2)()(")()( 2nn xRgRgxg

2)(")()( 211 nnnn egxgRgxRe

Ce

er

n

n

n

1limNewton’s method isquadratically convergent!

Recall that


Question from the last class

Is the initial solution important for theconvergence of fixed-point iteration?


Answer to the convergence problem offixed-point Iteration

|g’(R)|<1 is a necessary condition if |g’(R)|>1, the fixed-point iteration will diverge

even if the initial condition is very close to a rootsince the iteration will eventually reach the regioncausing divergence

Initial solution is important but less criticalThe algorithm may not converge if the initial

solution is far from the true solutionRecall the conditions that Newton’s method does

not converge?


Example: Newton’s Method for FindingComplex Roots

f(x) = x3 + 2x2 –x + 5


Start Newton’s method with a complex value

ix 10 ixf 52)( 0 ixf 103)(' 0

iii

ixfxf

xx 04587.1486238.010352

1)(')(

0

001

52)( 23 xxxxf 143)(' 2 xxxf

ixfxf

xx 23665.1448139.0)(')(

1

112


Newton’s Methods for Multiple Roots

Quadratically convergent for simple root,

Linearly convergent for multiple roots as

0)(')]('[

)(")()(' 0)(

2 Rgxf

xfxfxg Rf

00

)]('[)(")(

)(' 2 Rf

RfRfRg


Remedies for Multiple Roots withNewton’s method

If f(x) has a root of multiplicity k at x=R, wecan factor out (x-R)k from f(x) to get

With a slightly modified Newton’s method

It can be proved that andNewton’s method still converges quadratically

)()()( xQRxxf k

0)(' Rg k

)()(')(

1 nkn

nnn xg

xfxf

kxx


Remedies for Multiple Roots withNewton’s method (cont.)

In practice, we don’t know k in advanceRemedies

Try and errorDeflate f(x)

f(x)/(x-s) where s is an approximate RBe warned that an indeterminate form at x=R is

created


Systems of Nonlinear Equations Solving systems of nonlinear equations is

much more difficult than scalar case becauseWide variety of behavior is possible, so

determining existence and number of solutions orgood starting guess is much more complex

In general, there is no simple way to guaranteeconvergence to desired solution or to bracketsolution to produce absolutely safe method

Computational overhead increases rapidly withdimension of problem


Example: Systems in Two Dimensions

From Michael T. Heath


Newton’s Method In n dimensions, Newton’s method has form

where J(x) is Jacobian matrix of f

In practice, we do not explicitly invert J(xn),but instead solve linear system

for Newton step sn, then take as next iterate

)()( 1nnn1n xfxJxx

)()( nnn xfsxJ

nn1n sxx


Example: Newton’s Method

014

),(),(

)(2

22

21

212

211

1

xexx

xxfxxf

xxf

1

22)(

1

21

2

2

1

2

2

1

1

1

xexx

xf

xf

xf

xf

xJ

T7.110 x

0183.011.0

)( 0xf

17183.24.32

)( 0xJ

)()( 000 xfsxJ

0183.0

11.017183.2

4.320s

T]7298.10043.1[00 sxx1


Example: Newton’s Method

014

),(),(

)(2

22

21

212

211

1

xexx

xxfxxf

xxf

1

22)(

1

21

2

2

1

2

2

1

1

1

xexx

xf

xf

xf

xf

xJ

T7298.10043.11 x

49651.142653.8

)( 1 ee

xf

17300.24596.30086.2

)( 1xJ

)()( 111 xfsxJ

49651.142653.8

17300.24596.30086.2

1 ee

s

T]729637.1004169.1[112 sxx


Fixed-Point Iteration Fixed-point problem for is to find

vector x such thatCorresponding fixed-point iteration is

Converges if starts close enough to solution

)(xgx

)(1 nn xgx

nn RR :g

1))(( RJ

AA

RJ

matrixaofseigenvaluetheofnormcomplexmaximum:)(

solutiontrue:matrixJacobian:


Fixed-Point Iteration (cont.)Convergence rate is normally linear, with

constant C given by If then convergence rate is at least

quadratic, e.g., Newton’s method

nn RR :g

0)( RJ

))(( RJ


Example: Fixed-point Iteration

014

),(),(

)(2

22

21

212

211

1

xexx

xxfxxf

xxf

21

2

2

1

4)1ln(

xx

xx T7.110 x

1.00000000000000 -1.73205080756888

1 .00505253874238 -1.72912388057290

1 .00398063482304 -1.72974647995027

1 .00420874039761 -1.72961406264780

1 .00416023020985 -1.72964222660783


Next Monday

Introduction to Matlab/OctaveMatlab offers a student version with discounted

priceOctave is a shareware that has similar environment

as Matlab!

You are encouraged to bring your laptop withOctave or Matlab installed next Monday

Documents

Solving Nonlinear Equation - CAIG Lab - National Chiao ...caig.cs.nctu.edu.tw/course/NM/chap1.pdf · Solving Nonlinear Equation x 0 x 2 x 1 f ... System of n coupled nonlinear equations