Computational Computational Methods in PhysicsMethods in Physics

PHYS 3437 PHYS 3437Dr Rob ThackerDr Rob Thacker

Dept of Astronomy & Physics Dept of Astronomy & Physics (MM-301C)(MM-301C)

thacker@ap.smu.cathacker@ap.smu.ca

Today’s LectureToday’s Lecture

Functions and roots: Part IFunctions and roots: Part I PolynomialsPolynomials General rooting findingGeneral rooting finding

Bracketing & the bisection methodBracketing & the bisection method Newton-Raphson methodNewton-Raphson method Secant methodSecant method

Functions & rootsFunctions & roots

This is probably the first numerical task This is probably the first numerical task we encounter we encounter Many graphical calculators have built-in Many graphical calculators have built-in

routines for finding rootsroutines for finding roots Solving f(x)=0 is a problem frequently Solving f(x)=0 is a problem frequently

encountered in physicsencountered in physics First examine solutions for polynomials:First examine solutions for polynomials:

2

4 ; )( :2

rootonly theis ; )( :1

1 with 0)(

02112

102

001

0

aaaxxxaaxfn

axxaxfn

axaxfn

in

iin

Higher orders: cubicHigher orders: cubic

Cubic formula was discovered by Tartaglia Cubic formula was discovered by Tartaglia (1501-1545) but published by Cardan (1501-1545) but published by Cardan (~1530) (~1530) without permission!without permission!

1-i ; )31(2

1

182744

32

27

2

9

where )(3

1

)(3

1

)(3

1

))()(()( :2

012200

32

21

21

22

3/1

01232

221

222

21

32132

2103

i

aaaaaaaaa

iaaaa

ax

ax

ax

xxxxxxxxaxaaxfn

Higher orders: quartic & Higher orders: quartic & higherhigher

The general solution to a quartic was discovered by The general solution to a quartic was discovered by Ferrari (1545) but is Ferrari (1545) but is extremelyextremely lengthy lengthy

Evariste Galois (1811-1832) showed that there is no Evariste Galois (1811-1832) showed that there is no general solution to a quintic & higher formulasgeneral solution to a quintic & higher formulas

Remember certain higher order equations may be Remember certain higher order equations may be rewritten as quadratics, cubics in xrewritten as quadratics, cubics in x22 (say) e.g. (say) e.g.

What is the best approach for polynomials?What is the best approach for polynomials? Always best to reduce to simplest form by dividing out Always best to reduce to simplest form by dividing out

factorsfactors If coefficients sum to zero then x=1 is a solution and (x-1) If coefficients sum to zero then x=1 is a solution and (x-1)

can be factored to give a (x-1)*(polynomial order n-1)can be factored to give a (x-1)*(polynomial order n-1) Can you think why?Can you think why?

You may be able to repeatedly factor larger polynomials by You may be able to repeatedly factor larger polynomials by inspection inspection Only useful for a single set of coefficients thoughOnly useful for a single set of coefficients though

1)(3)(13 22224 xxxx

ExampleExample Consider f(x)=xConsider f(x)=x55+4x+4x44-10x-10x22--

x+6=0x+6=0 Firstly sum of coefficients Firstly sum of coefficients

is 0, so x=1 is a solution – is 0, so x=1 is a solution – factor (x-1) by long division factor (x-1) by long division of the polynomialof the polynomial

Hence we find thatHence we find thatf(x)=(x-1)(xf(x)=(x-1)(x44+5x+5x33+5x+5x22-5x-6)-5x-6) Notice that the coefficients in Notice that the coefficients in

the second bracket sum to 0…the second bracket sum to 0… In fact we can show that In fact we can show that

further division givesfurther division givesf(x)=(x-1)f(x)=(x-1)22(x+1)(x+2)(x+3)(x+1)(x+2)(x+3) 66

55

5

5 5

105

55

05

6 5 5561004)1(

2

2

23

23

34

34

45

234

2345

x

xx

xx

xx

xx

xx

xx

xx

xxxxxxxxxx

So regardless of whether an analytic general solution exists, we may still be able tofind roots without using numerical root finders…

Advice on using the general Advice on using the general quadratic formulaquadratic formula

For the quadratic f(x)=xFor the quadratic f(x)=x22+a+a11x+ax+a0 0 the the product of the roots must be aproduct of the roots must be a00 since since f(x)=(x-xf(x)=(x-x11)(x-x)(x-x22) where x) where x11 & x & x22 are the are the rootsroots

Hence xHence x22=a=a00/x/x1 1 soso given one root, given one root, provided aprovided a00≠0, we can then quickly find ≠0, we can then quickly find the other rootthe other root

Using the manipulation discussed in Using the manipulation discussed in lecture 3lecture 3

dx

dax

aaaad

aaa

a

aaa

aaaaaax

2

01

02111

0211

0

0211

02110

211

1

/

4)sgn(2

1

defineThen

4

2

4

4

2

4

sgn(a1) is a function that is 1if a1 is positive, and -1 if a1 isnegative. This ensures that thediscriminant is added with thecorrect sign.

If a0=0 thenwe have divideby zero here.

More general root More general root findingfinding

For a more general function, say For a more general function, say f(x)=cos x – x the roots are non-f(x)=cos x – x the roots are non-trivialtrivial

1.0

0.8

h(x)=x

g(x)=cos x

In this case the solutionis given where g(x)=h(x)

Graphing actually helps agreat deal since you can seethere must only be oneroot. In this case around 0.8.

It is extremely beneficial to knowwhat a given function will looklike (approximately) – helps youto figure out whether your solutionsare correct or not.

Iterative root finding Iterative root finding methodsmethods

It is clearly useful to construct numerical algorithms It is clearly useful to construct numerical algorithms that will take any input f(x) and search for solutionsthat will take any input f(x) and search for solutions However, there is no perfect algorithm for a given function However, there is no perfect algorithm for a given function

f(x)f(x) Some converge fast for certain f(x) but slowly for othersSome converge fast for certain f(x) but slowly for others Certain f(x) may not yield numerical solutionsCertain f(x) may not yield numerical solutions

Above all you should not use algorithms as black Above all you should not use algorithms as black boxesboxes You need to know what the limitations are!You need to know what the limitations are!

We shall look at the following algorithmsWe shall look at the following algorithms BisectionBisection Newton-RaphsonNewton-Raphson Secant MethodSecant Method Muller’s MethodMuller’s Method

Bracketing & BisectionBracketing & Bisection Bracketing is a beautifully Bracketing is a beautifully

simple ideasimple idea Anywhere region bounded by Anywhere region bounded by

xxll and x and xuu, such that the sign , such that the sign of a function f(x) changes of a function f(x) changes between f(xbetween f(xll) and f(x) and f(xuu), there ), there must be must be at leastat least one root one root This is provided that the This is provided that the

function does not have any function does not have any infinities within the bracket – infinities within the bracket – f(x) must be continuous within f(x) must be continuous within the regionthe region

The root is said to be The root is said to be bracketedbracketed by the interval by the interval (x(xll,x,xuu))

f(x)

xl

xu

x

Other possibilitiesOther possibilities Even if a region is Even if a region is

bracketed by two bracketed by two positive (or two positive (or two negative) values a negative) values a root may still existroot may still exist

There could even There could even be 2n roots in this be 2n roots in this region!region! Where n is a Where n is a

natural numbernatural number

xl xuxl xu

x2 x3

Pathological behaviour Pathological behaviour occursoccurs

This is a graph of sin(1/x). As x→0 the function develops an infinite number of roots.

But don’t worry! This is not a course in analysis…

Bracketing singularitiesBracketing singularities

f(x)=1/(x-1) is not continuous within the bracketed region [0.5,1.5].Although f(0.5)<0 f(x)=1/(x-1) is not continuous within the bracketed region [0.5,1.5].Although f(0.5)<0 and f(1.5)>0 there is no root in this bracket (regardless of the fact that the plotting and f(1.5)>0 there is no root in this bracket (regardless of the fact that the plotting program connects the segments) f(x) is not continuous within the bracket.program connects the segments) f(x) is not continuous within the bracket.

Root finding by bisectionRoot finding by bisection Step 1Step 1: Given we : Given we

know that there exists know that there exists an f(xan f(x00)=0 solution, )=0 solution, choose xchoose xll and x and xuu as as the brackets for the the brackets for the rootroot Since the Since the signs ofsigns of f(x f(xll) )

& f(x& f(xuu) ) must be must be differentdifferent, we must , we must have f(xhave f(xll)×f(x)×f(xuu) < 0 ) < 0

This step may be non-This step may be non-trivial – we’ll come trivial – we’ll come back to itback to it

xl

xu

For example, for f(x)=cos x –x we know that 0<x0</2

f(x)

Bisection: Find the mid-Bisection: Find the mid-pointpoint

Step 2: Let Step 2: Let xxmm=0.5(x=0.5(xll+x+xuu)) the mid point between the mid point between

the two bracketsthe two brackets This point must be This point must be

closer to the root than closer to the root than one of xone of xll or x or xuu

The next step is to The next step is to select the new select the new bracket from xbracket from xll,x,xmm,x,xuu

xl

xu

xm

Bisection: Find the new Bisection: Find the new bracketbracket

Step 3: Step 3: If f(xIf f(xll)×f(x)×f(xmm)<0)<0

root lies between xroot lies between xll and xand xmm

Set xSet xll=x=xll ; x ; xuu=x=xmm

If f(xIf f(xll)×f(x)×f(xmm)>0)>0 root lies between xroot lies between xmm

and xand xuu

Set xSet xll=x=xmm; x; xuu=x=xuu

If f(xIf f(xll)×f(x)×f(xmm)=0)=0 root is xroot is xmm you can stop you can stop

xl

xu

xm

Final Step in BisectionFinal Step in Bisection Step 4: Step 4: Test accuracyTest accuracy (we want to (we want to

stop at some point)stop at some point) xxmm is the best guess at this stage is the best guess at this stage Absolute error is |xAbsolute error is |xmm-x-x00| but we don’t | but we don’t

know what xknow what x00 is! is! Error estimate: Error estimate: |x|xmm

nn-x-xmmn-1n-1| where n | where n

corresponds to nth iterationcorresponds to nth iteration Check whether Check whether where where is the is the

tolerancetolerance that you have to decide for that you have to decide for yourself at the yourself at the startstart of the root finding of the root finding If If then stopthen stop If If then return to step 1 then return to step 1

ExampleExample Let’s consider f(x)=cos x - xLet’s consider f(x)=cos x - x Pick the initial bracket as xPick the initial bracket as x ll=0 and x=0 and xuu==/2/2

f(0)=1, f(f(0)=1, f(/2)=-/2)=-/2 – check this satisfies /2 – check this satisfies f(0)×f(f(0)×f(/2)<0 (/2)<0 (yes -yes - so good to start from here) so good to start from here)

First bisection: First bisection: xxmm=0.5(x=0.5(xll+x+xuu)=)=/4=0.785398/4=0.785398 f(xf(xll

(1)(1))=f(0)=1)=f(0)=1 f(xf(xmm

(1)(1))=f()=f(/4)=-0.078/4)=-0.078 f(xf(xuu

(1)(1))=f()=f(/2)=-/2)=-/2/2 So select xSo select xll

(1) (1) && xxmm(1)(1) as the new bracket as the new bracket

f(xf(xll(1)(1))×f(x)×f(xmm

(1)(1))<0)<0

Repeat…Repeat…

After eleventh bisection: After eleventh bisection: xxmm

(11)(11)=963=963/4096=0.738612/4096=0.738612 After twelfth bisection: After twelfth bisection:

xxmm(12)(12)=1927=1927/8192=0.738995/8192=0.738995

So after first 12 bisections we have So after first 12 bisections we have only got the first three decimal places only got the first three decimal places correctcorrect

Typically to achieve 9 decimal places of Typically to achieve 9 decimal places of accuracy will require 30-40 stepsaccuracy will require 30-40 steps

Pros & cons of bisectionPros & cons of bisection Pro: Bisection is robustPro: Bisection is robust

You will You will alwaysalways find a root find a root For this fact alone it should be taken seriouslyFor this fact alone it should be taken seriously

Pro: can be programmed very Pro: can be programmed very straightforwardlystraightforwardly

Con: bisection converges slowly…Con: bisection converges slowly… Width of the bracket: xWidth of the bracket: xuu

nn-x-xllnn=e=enn=e=en-1n-1/2/2

Width halves at each stageWidth halves at each stage We know in advance if the initial bracket is width We know in advance if the initial bracket is width

ee00 after n steps the bracket will have width e after n steps the bracket will have width e00/2/2nn

If we have a specified tolerance If we have a specified tolerance then this will be then this will be achieved when n=logachieved when n=log22(e(e00//))

Bisection is said to converge linearly since eBisection is said to converge linearly since en+1n+1eenn

Newton-Raphson MethodNewton-Raphson Method For many numerical methods, the underlying algorithms For many numerical methods, the underlying algorithms

begins with a Taylor expansionbegins with a Taylor expansion Consider the Taylor expansion of f(x) around a point xConsider the Taylor expansion of f(x) around a point x00

If x is a root thenIf x is a root then

This formula is applicable provided f’(xThis formula is applicable provided f’(x00) is not an ) is not an extremum (f’(xextremum (f’(x00)≠0)))≠0))

...)(')()()( 000 xfxxxfxf

)('

)(

)(')()(0

0

00

000

xf

xfxx

xfxxxf

Applying the formulaApplying the formula If we interpret xIf we interpret x00 to be the current guess we to be the current guess we

use the formula to give a better estimate for use the formula to give a better estimate for the root, xthe root, x We can repeat this process iteratively to get better We can repeat this process iteratively to get better

and better approximations to the root and better approximations to the root We may even get the exact answer if we are luckyWe may even get the exact answer if we are lucky

Take f(x)=cos x – x, then f’(x)=-sin x -1Take f(x)=cos x – x, then f’(x)=-sin x -1 It helps to make a good guess for the first It helps to make a good guess for the first

root, so we’ll take xroot, so we’ll take x00(1)(1)==/4=0.785398 /4=0.785398

The (1) signifies the first in a series of guessesThe (1) signifies the first in a series of guesses f(xf(x00

(1)(1))=-0.0782914 )=-0.0782914 f’(xf’(x00

(1)(1))=-1.7071068)=-1.7071068

Steps in the iterationSteps in the iteration

Plugging in the values for f(xPlugging in the values for f(x00(1)(1)), f’(x), f’(x00

(1)(1)) we get ) we get

We repeat the whole step again, with the values We repeat the whole step again, with the values for xfor x00

(2)(2)

At this stage f(xAt this stage f(x00(3)(3))=0.0000002, so we are )=0.0000002, so we are

already very close to the correct answer – after already very close to the correct answer – after only two iterationsonly two iterations

It helps to see what is happening geometrically…It helps to see what is happening geometrically…

739536.07071068.1

0782914.0785398.0

)('

)()1(

0

)1(0)1(

0)2(

0 xf

xfxx

739085.0673945.1

0007546.0739536.0

)('

)()2(

0

)2(0)2(

0)3(

0 xf

xfxx

Geometric Interpretation Geometric Interpretation of N-R of N-R

The general step in The general step in the iteration the iteration process isprocess is

)('

)()(

)()()1(

n

nnn

xf

xfxx

f(x)

x(n) xi=x(n+1)

x(n+2)

x(n+3)

Draw out a line from f(x(n)) with slopef’(x(n)) – equivalent to a line y=m(x-x(n))+bwhere m=f’(x(n)) and b=f(x(n)) – check it!The next step in the iteration is when y=0.

So N-R “spirals in” onthe solution

Problems with N-RProblems with N-R

The primary problem with N-R is The primary problem with N-R is that if you are too close to an that if you are too close to an extremum the derivative f’(xextremum the derivative f’(x(n)(n)) can ) can become very shallow:become very shallow:

f(x)

x(n)

xuxl

In this case the root was originallybracketed by (xl,xu), but the valueof x(n+1) will lie outside that range

to x(n+1)

NR-Bisection HybridNR-Bisection Hybrid Let’s combine the two methodsLet’s combine the two methods

Good convergence speed from NRGood convergence speed from NR Guaranteed convergence from bisectionGuaranteed convergence from bisection

Start with xStart with xll<x<x(n)(n)<x<xuu as being the current best as being the current best guess within the bracketguess within the bracket

LetLet

If xIf xll<x<xNRNR(n+1)(n+1)<x<xuu then take an NR step then take an NR step

Else set xElse set xNRNR(n+1)(n+1)=x=xmm

(n+1)(n+1)=0.5(x=0.5(xll+x+xuu), check and update x), check and update xll and xand xuu values for the new bracket values for the new bracket

i.e.i.e. we throw away any bad NR steps and do a bisection we throw away any bad NR steps and do a bisection step insteadstep instead

)('

)()(

)()()1(

n

nnn

NR xf

xfxx

Example: Full NR-Bisection Example: Full NR-Bisection algorithmalgorithm

Let’s look at how we might design the Let’s look at how we might design the algorithm in detailalgorithm in detail

Initialization:Initialization: Given xGiven xll and x and xuu from user input evaluate f(x from user input evaluate f(xll) & f(x) & f(xuu))

Test that xTest that xll≠x≠xu u – if they are equal report to user– if they are equal report to user Ensure root is properly bracketed:Ensure root is properly bracketed:

Test f(xTest f(xll)×f(x)×f(xuu)<0 – if not report to user)<0 – if not report to user Initialize xInitialize x(n)(n) by testing values of f(x by testing values of f(xll) & f(x) & f(xuu))

If |f(xIf |f(xll)| < |f(x)| < |f(xuu)| then x)| then x(n)(n)=x=xll and set f(x and set f(x(n)(n))= f(x)= f(xll) ) else let xelse let x(n)(n)=x=xu u and set f(xand set f(x(n)(n))=f(x)=f(xuu)) Evaluate the derivative f’(xEvaluate the derivative f’(x(n)(n)))

Set the counter for the number of steps to zero: Set the counter for the number of steps to zero: icount=0icount=0

Full NR-Bisection Full NR-Bisection algorithm cont.algorithm cont.

(1) Perform NR-bisection(1) Perform NR-bisection Increment icountIncrement icount Evaluate xEvaluate xNRNR

(n+1)(n+1)=x=x(n)(n)-f(x-f(x(n)(n))/f’(x)/f’(x(n)(n))) If (xIf (xNRNR

(n+1)(n+1)<x<xll) or (x) or (xNRNR(n+1)(n+1)>x>xuu) then require bisection) then require bisection

Set xSet xNRNR(n+1)(n+1)=0.5(x=0.5(xll+x+xuu))

Calculate error estimate: Calculate error estimate: =|x=|x(n+1)(n+1)-x-x(n)(n)|| Set x(n)=x(n+1)Set x(n)=x(n+1) Test errorTest error

If |If |x(n)|< tolerance go to (2)x(n)|< tolerance go to (2) Test number of iterationsTest number of iterations

If icount>max # iterations then go to (3)If icount>max # iterations then go to (3) Prepare for next iterationPrepare for next iteration

Set value of f(x(n))Set value of f(x(n)) Set value of f’(x(n))Set value of f’(x(n)) Update the brackets using Step 3 of the bisection algorithmUpdate the brackets using Step 3 of the bisection algorithm

Go to 1Go to 1

Final parts of the Final parts of the algorithmalgorithm

(2) Report the root and stop(2) Report the root and stop (3) Report current best guess for root. Inform (3) Report current best guess for root. Inform

user max number of iterations has been user max number of iterations has been exceededexceeded

While it is tempting to think that these While it is tempting to think that these reporting stages are pointless verbiage – they reporting stages are pointless verbiage – they absolutely aren’t!absolutely aren’t! Clear statements of the program state will help Clear statements of the program state will help

you to debug tremendouslyyou to debug tremendously Get in the habit now!Get in the habit now!

Secant Method - Secant Method - PreliminariesPreliminaries

One of the key issues in N-R is that you need the derivativeOne of the key issues in N-R is that you need the derivative May not have it if the function is defined numericallyMay not have it if the function is defined numerically Derivative could be extremely hard to calculateDerivative could be extremely hard to calculate

You can get an estimate of the local slope using values of xYou can get an estimate of the local slope using values of x(n)(n),x,x(n-(n-

1)1), f(x, f(x(n)(n)) and f(x) and f(x(n-1)(n-1)))

)1()(

)1()()( )()()('

nn

nnn

xx

xfxf

x

yxf

f(x)

x(n)

x(n-1)

x=x(n)-x(n-1)

y=f(x(n))-f(x(n-1))

Estimate of functionslope

Secant MethodSecant Method

So once you initially select a pair of points So once you initially select a pair of points bracketing the root (xbracketing the root (x(n-1)(n-1), x, x(n)(n)), x), x(n+1)(n+1) is given is given byby

You then iterate to a specific tolerance You then iterate to a specific tolerance value exactly as in NRvalue exactly as in NR

This method can converge almost as quickly This method can converge almost as quickly as NRas NR

)()()(

)1()(

)1()()()()1(

nn

nnnnn

xfxf

xxxfxx

Issues with the Secant Issues with the Secant MethodMethod

Given the initial values xGiven the initial values xll(=x(=x(n-1)(n-1)),x),xuu(=x(=x(n)(n)) ) that bracket the root the calculation of that bracket the root the calculation of xx(n+1)(n+1) is is locallocal Easy to program, all values are easily storedEasy to program, all values are easily stored

Difficulty is with finding suitable xDifficulty is with finding suitable xll,x,xuu since this is a global problemsince this is a global problem Determined by f(x) over its entire rangeDetermined by f(x) over its entire range Consequently difficult to program (no perfect Consequently difficult to program (no perfect

method)method)

Case where Secant will Case where Secant will be poorbe poor

1 3

2

This is actually a difficult function todeal with for many root finders…

Estimated slope is prettymuch always 45 degreesuntil you get close to theRoot.

Bracket finding Bracket finding StrategiesStrategies

Graph the functionGraph the function May require 1000’s of points and thus May require 1000’s of points and thus

function calls to determine visually where function calls to determine visually where the function crosses the x-axisthe function crosses the x-axis

““Exhaustive searching” – a global Exhaustive searching” – a global bracket finderbracket finder Looks for changes in the sign of f(x)Looks for changes in the sign of f(x) Need small steps so that no roots are missed Need small steps so that no roots are missed Need still to take large enough steps that Need still to take large enough steps that

this doesn’t take foreverthis doesn’t take forever

SummarySummary When dealing with high order polynomials don’t When dealing with high order polynomials don’t

rule out the possibility of being able to factorize rule out the possibility of being able to factorize by handby hand

For general functions, bisection is simple but For general functions, bisection is simple but slowslow Nonetheless the fact that it is guaranteed to Nonetheless the fact that it is guaranteed to

converge from good starting points is converge from good starting points is extremely usefulextremely useful

Newton-Raphson can have excellent convergence Newton-Raphson can have excellent convergence propertiesproperties Watch out for poor behaviour around Watch out for poor behaviour around

extremumsextremums Secant method can be used in place of NR when Secant method can be used in place of NR when

you cannot calculate the derivativeyou cannot calculate the derivative

Next LectureNext Lecture

More on Global bracket findingMore on Global bracket finding Müller-Brent methodMüller-Brent method ExamplesExamples