Num Course

8/10/2019 Num Course

1/53

Introduction to Numerical Methods

Dr Jamieson Christie

October 1, 2009

Contents

0.1 Syllabus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

0.2 Useful Linux commands . . . . . . . . . . . . . . . . . . . . . . . 3

1 Introduction 7

1.1 Introduction to Linux . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.1 The command line . . . . . . . . . . . . . . . . . . . . . . 7

1.1.2 Files and directories . . . . . . . . . . . . . . . . . . . . . 7

1.1.3 Creating files . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2 Introduction to FORTRAN . . . . . . . . . . . . . . . . . . . . . 9

1.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2.2 Hello World! . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2.3 A second program . . . . . . . . . . . . . . . . . . . . . . 10

1.2.4 Real numbers . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.2.5 if statements and logical constructs . . . . . . . . . . . . 13

1.2.6 Mathematical functions . . . . . . . . . . . . . . . . . . . 15

1.2.7 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.2.8 Input and output: reading and writing data . . . . . . . . 17

1.3 Introduction tognuplot . . . . . . . . . . . . . . . . . . . . . . . 18

1.4 Representation of numbers . . . . . . . . . . . . . . . . . . . . . . 20

1.4.1 Integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

1.4.2 Floating-point numbers . . . . . . . . . . . . . . . . . . . 21

1


2/53

2 Finding zeroes of a function 23

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.2 bisection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.3 The regula falsi method . . . . . . . . . . . . . . . . . . . . . . . 26

2.4 The secant method . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.5 Newtons method . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.6 Iterative procedures . . . . . . . . . . . . . . . . . . . . . . . . . 30

3 Linear algebra 31

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.2 Elementary operations . . . . . . . . . . . . . . . . . . . . . . . . 313.3 Solving systems of linear equations . . . . . . . . . . . . . . . . . 32

3.3.1 Elementary operations that do not change the solution . . 32

3.3.2 Gaussian elimination . . . . . . . . . . . . . . . . . . . . . 33

3.3.3 Computational implementation . . . . . . . . . . . . . . . 34

3.3.4 Pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.3.5 Singular matrices and conditioning . . . . . . . . . . . . . 37

3.3.6 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.3.7 Finding the inverse . . . . . . . . . . . . . . . . . . . . . . 38

3.3.8 Special matrices: tridiagonal, sparse etc. . . . . . . . . . . 39

3.3.9 Iterative methods . . . . . . . . . . . . . . . . . . . . . . . 39

4 Numerical integration 39

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2 Trapezium rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.3 Function interpolation . . . . . . . . . . . . . . . . . . . . . . . . 42

4.4 Simpsons rule and more complicated approximations . . . . . . . 43

4.5 Varying strip size . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.6 Errors in numerical integration . . . . . . . . . . . . . . . . . . . 45

2


3/53

5 Numerical differentiation 45

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455.2 The first guess: the forward-difference approximation . . . . . . . 45

5.3 The centred-difference approximation . . . . . . . . . . . . . . . . 46

5.4 Higher-order derivatives . . . . . . . . . . . . . . . . . . . . . . . 49

5.5 Differentiation by function interpolation . . . . . . . . . . . . . . 49

5.6 Richardson extrapolation . . . . . . . . . . . . . . . . . . . . . . 50

Introduction

These notes were originally written in the summer of 2007 for the introductorypart of the Introduction to Numerical Methods course given in the CondensedMatter Diploma course at the Abdus Salam International Centre for TheoreticalPhysics (ICTP). They have not been changed since, and so many not be suitablefor any other use. For example, there are occasional references to the ICTPcomputer set-up.

The notes were intended to be used for students who had not necessarily had anyexperience of programming or computers, and so they start from a very basiclevel. They formed a ten-lecture course, where each lecture took 90 minutes.

If you find any errors while reading these notes, I would be very grateful if youlet me know.

Provided this notice is preserved, you may freely distribute all or part of thesenotes for personal or academic use. You may not: remove this notice, make anycommercial use of these notes, modify these notes or distribute any modifiedcopy, make and/or distribute a larger work of which these notes form a part.

Jamieson Christie

[email protected]

October 2009

3


4/53

Syllabus

INTRODUCTION Files and directories in Linux; creating files; compilingand executing FORTRAN code; variable declarations; logical constructs; doloops; mathematical functions; matrices; plotting data withgnuplot; input andoutput; representing numbers

ROOT FINDING Bisection; regula falsi; secant method; Newtons method;use of functions; functional iteration

LINEAR ALGEBRA Matrix algebra; solving systems of equations NUMERICAL INTEGRATION Trapezoid rule; Simpsons rule NUMERICAL DIFFERENTIATION Forward and centred difference methods

4


5/53

Useful Linux commands

mkdir

Create a directory called name with mkdir name

[jchristi@condense-38 ~]$ mkdir name

cd

Change to a directory calledname with cd name

[jchristi@condense-38 ~]$ cd name

[jchristi@condense-38 ~/name]$

ls

List the contents of the current directory with ls.

[jchristi@condense-38 ~/name]$ ls

[jchristi@condense-38 ~/name]$

pwd

Find out which directory you are in with pwd.

[jchristi@condense-38 ~/name]$ pwd

/afs/ictp/home/j/jchristi/name

cd again

Move to the parent directory with cd ...

[jchristi@condense-38 ~/name]$ cd ..

[jchristi@condense-38 ~]$

From any directory, you can return to your home directory by using cd without

the name of any directory.

[jchristi@condense-38 Overall]$ pwd

/afs/ictp/home/j/jchristi/MDprogram/Dec1200K/Overall

[jchristi@condense-38 Overall]$ cd


5


6/53

mv

Change the name of a directory or file withmv. For instance, to change the nameof the directory name to programs, go to the parent directory of /tt name, andtype

[jchristi@condense-38 ~]$ mv name programs

cp

To copy a file, use cp. For instance, to make a copy offile1calledfile2, type

[jchristi@condense-38 ~]$ cp file1 file2

rm

To delete a file, userm. Once deleted, the file cannot be recovered. For example,to delete a file file1, type

[jchristi@condense-38 ~]$ rm file1

rmdir

To remove an empty directory, use rmdir. If you want to remove a directorywith files in, use rm -r. Be careful: once deleted, the files cannot be recovered.

[jchristi@condense-38 ~]$ rm dir1

[jchristi@condense-38 ~]$ rmdir dir1

rmdir: dir1: Directory not empty

[jchristi@condense-38 ~]$ rm -r dir1


man

The most useful command! To find out about any command, type man, thenthe name of the command. For example, for more information on the commandls, type

[jchristi@condense-38 ~]$ man ls

6


7/53

more

To look in a text file one screen at a time, you can use the morecommand.

[jchristi@condense-38 ~]$ more filename

less

A more powerful command to look at text files is less, which allows us to movebackwards and forwards in the file.

[jchristi@condense-38 ~]$ less filename

head

To look at the start of a file, use head. By default, it shows the first 10 lines ofthe file, use head -n 20if you want to see the first 20, for example.

[jchristi@condense-38 ~]$ head -n 2 sos.cpp

#include

#include

tail

The opposite command is tail, which looks at the end of a file. The default isthe end 10 lines, use tail -n 20if you want to see the end 20, for example.

[jchristi@condense-38 ~]$ tail -n 2 sos.cpp

return 0;

}

grep

The command grepallows us to search inside files for a pattern. For example,say I have files called file1 and file2 and I want to find out which of themcontains the word Diploma. I would type

[jchristi@condense-38 ~]$ grep Diploma file1 file2

file2:Diploma 2007-08, which will be

The program returns the whole line containing the word.

7


8/53

wc

The commandwc counts the number of lines, words and characters in a file. Forexample,

[jchristi@condense-38 ~]$ wc *cpp

32 90 669 adjust2.cpp

29 86 590 adjust.cpp

61 176 1459 total

The first figure is the number of lines, the second figure is the number of words,and the third is the number of characters.

ls -l

Although there are lots of options for ls, one of the most useful is ls -l, whichshows lots of information about the files in the directory. For example,

[jchristi@condense-38 ~]$ ls -l *cpp

-rw-r--r-- 1 jchristi smr1694 669 May 24 15:46 adjust2.cpp

-rw-r--r-- 1 jchristi smr1694 590 Dec 11 2006 adjust.cpp

Of most interest are the permissions, the owner (jchristi), the size of the file(here in kB), and the date and time it was last modified.

*

The * acts as a wild card. For example, ls *cppwill last all files which end incpp, whereas ls cpp*will last all files which begin in cpp.

Tab completion

If you only know part of a filename or directory, by pressing the Tab key, youcan obtain the rest of the name, providing that it is unique. This speeds uptyping considerably.

8


9/53

1 Introduction

1.1 Introduction to Linux

1.1.1 The command line

The first part of this course is an introduction to the Linux operating system.This is the operating system in which a lot of scientific computing is done. Allthe computers in ICTP can boot into either the Linux1 or Windows operatingsystem. When the machine starts up, choose linux, rather than windows fromthe first menu. Eventually, you will be asked for your username and password.When you provide this, you will log in to Linux.

Once you have logged into the computer and Linux is running, you will needto open a terminal. In this course, we will be working mainly from the com-mand line. This is a plain-text interface where we can type commands directly.There are hundreds of possible commands and processes we can do from thecommand line; Linux gives its users a lot of control over the computer and itsoperating system. In this course I will concentrate on the essentials for scientificcomputing.

A terminal can be opened with a program called XTerm or KTerm or Terminalor something similar. When you do this, you will see something like this:


This is called the command line. You will see a slightly different command linewith your username and the name of the computer you are using (jchristi ismy username, and condense-38 is the name of the desktop computer in myoffice, where I was writing these notes).

When you have this line, you can issue commands by typing their name, anyarguments that they take, and pressing the Enter key.

1.1.2 Files and directories

When you open a terminal, you are put in your home directory. Each directorycan contain files and other directories. You can create new directories with thecommand mkdir, followed by the name of the directory to be created. Youshould name the directory with a clear idea of what you can expect to find

in it. This makes for much easier filing and retrieval. For example, in myhome directory, I have directories called applications, backups, Diploma,Documents, MDprogram, Papers, Pictures, private, among others. Manyof these directories have sub-directories inside them, for example, the Diploma

1There are many different types of Linux, and I will not go into the differences. In this

course, we will use whichever type happens to be installed on the ICTP computers.

9


10/53

directory has a directory called NumericalMethods, which holds all my notesfor this course. Note that you cannot easily use a space in a directory name,

which is why I called it NumericalMethods rather than Numerical Methods.

You can use the command cd followed by a directory name to change to a newdirectory, as well as cd .., which will move you one level back up the directorytree, and cd without a specifying directory, which returns you back to yourhome directory.

The command ls lists the files in the current directory. Typing ls -l givesmore information on the files: the file mode, number of links, owner and group,file size, and time of last modification.

1.1.3 Creating files

There are very many ways to create text files. This course mainly covers pro-gramming, so it makes sense to choose an editor which offers some support forprogramming. My favourite is called emacs, which is widely used. It can becalled from the command line by typing:

[jchristi@condense-38 ~]$ emacs &

The & means that emacs should be run in the background. This means thatcontrol will be returned to the command line. emacs will open a new windowin which you can type.

When you have finished typing in emacs, you can save the file by executingthe command Ctrl-x Ctrl-s. The notationCtrl-x means that the Ctrl keyshould be held down, and the x key pressed while the Ctrl key is still beingheld. To close emacs you use the command Ctrl-x Ctrl-c. To open emacsand look at a specific file, type emacs filename &

Once you have created and saved a file, you can return to the command line,type ls, and see that it will be listed there.

Sometimes you will not need to modify a file, but just to look at it. Thecommandsmore and less do this: more allows us to go page by page forwardsthrough the file, whereaslessallows us to go forward and backward. If you justwant to look at the first few lines of a file, you can use the headcommand, andif you just want to look at the last few lines, you can use the tailcommand.

If you have many files, and cant remember which one contains a certain wordor phrase, the grepcommand will search the files for a given phrase.

The numbers of lines, words and characters in a certain file can be obtainedwith the wc command.

Here, I have listed only a small subset of the commands, essentially, enough toget you started. There are very many others, and you can ask us if there issomething specific that you want to do.

10


11/53

1.2 Introduction to FORTRAN

1.2.1 Introduction

In this course, you will learn how to program a computer to perform usefulscientific tasks. In order to do this, you will write code in a language calledFORTRAN. There are hundreds of different programming languages available.For our purposes, FORTRAN has two main benefits: it is easy to learn, and alot of scientific code is written in it.

After having written the code in FORTRAN, you will need to compileit. To dothis you will use a program called a compiler, which translates the FORTRANcode into machine code. Once we have done this, we will then create an exe-cutable file, which the computer can run. Usually, these two steps are done atonce.

1.2.2 Hello World!

Your first program will be very simple: you will write code which will print HelloWorld! to the screen. Open emacswith the command emacs helloworld.f90&

helloworld.f90does not yet exist, so emacs shows you a blank page.

The FORTRAN code is as follows:

program hello

write(*,*)Hello World!

end program

Type this code exactly as written into emacs and save the file with Ctrl-xCtrl-s.

The program starts with the line program hello. This tells the compiler thatthe program starts there, and is called hello. You can name the program asyou wish. The command write(*,*) tells the compiler to write to the screenwhatever follows between the quotation marks. The program then ends withend programso that the compiler knows the program has ended.

There are many different compilers that can be used. You can use g95 whichshould be installed on the ICTP machines, and is available for free if not. Inorder to compile the program, you must type the following on the commandline, after having saved the file:

[jchristi@condense-38 ~]$ g95 helloworld.f90 -o helloworld.x

This tells the compiler (g95) to compile the file helloworld.f90, and create anexecutable program calledhelloworld.x. If this step is carried out successfully,then the executable helloworld.xwill appear when the files are listed with ls.

11


12/53

You can then run the program with the command ./helloworld.xThe ./ before the program name tells the computer that the executable file is

in the current directory.

[jchristi@condense-38 ~]$ ./helloworld.x

Hello World!

If all works well, you should see that the program has written Hello World! tothe screen, just as you wanted.

Pause here and make sure that you understand what you have done. Perhapschange the text to print some other message to the screen (perhaps Hello world!in your own language), or more than one. Change the name of the executablefile.

1.2.3 A second program

Now, you try a more complicated example. You will write a program to computethe first 100 square numbers. Use emacsto open a new file called squares.f90,and type the following exactly as written:

program squares

implicit none

! This program computes the first 100 squares

! and prints them to the screen

integer i

do i=1,100

write(*,*)i,i*i

enddo

end program

Dont worry about what the program does just yet, just compile it with

[jchristi@condense-38 ~]$ g95 squares.f90 -o squares.x

and run with

[jchristi@condense-38 ~]$ ./squares.x

You will see that it prints to the screen, the numbers from 1 to 100, and theirsquares.

12


13/53

When you want the computer to do a number of nearly identical things, youcan use a do loop, such as the one in the program above. The code inside the

doloop, which is everything between the do line and theenddoline, is executeda number of times. This number is set by the first line of the loop. In thiscase, the variable i is set equal to 1 to begin. Then the code inside the loop isexecuted once. Then the program returns to the start of the loop, and adds 1to i. This continues to happen until i is equal to 100, in our example. Thismeans that the code inside the loop is executed 100 times.

The code inside the loop is a simple write command such as you have alreadyused. It first writes i to the screen, and then the square of i, which in FOR-TRAN is written as i*i. Since i increases by 1 each time the loop is executed,the program writes the first 100 numbers and their squares to the screen.

There is another new feature in this program. For the first time, you are usinga variable, which you have called i. The compiler needs to know what i is sothat it knows how much memory to allocate, for example. At the beginningof the program, you must declare the variable: tell the compiler what type ofvariable it is. Since here i is an integer, you include a line integer i. It ispossible for FORTRAN to use variables that have not been declared, however,this is almost always a disaster, and offers a benefit only in a very small numberof cases. To turn off this option, and force FORTRAN to accept only declaredvariables, the command implicit noneis included at the start of the program,before the declarations. This line should always be included in your programs.

The final new feature is the comments included in the program. The programsyou have written so far have been very simple, but many programs are extremelycomplicated and would be difficult to understand just from the code alone. Soyou can write comments to explain to yourself or future programmers what

parts of the code are doing. These comments begin with ! and the rest of theline then forms the comment. The compiler ignores the comments.

Again, you should pause here and make sure you understand what this programis doing. You can try changing the start and end points of the doloop, or changethe values written to the screen, for instance, writing the cube with i*i*i ormore complicated expressions like i*i - 5*i + 6

You can change the step size of the loop, by including a third number in thefirst line of the do loop. For example, do i=1,100,2 will execute a loop andthen add 2 to i. If you want i to decrease, you should use a negative numberas follows do i=100,1,-1. Experiment with these options.

In some cases, you will not want to have the output written to the screen, but

rather put into a file. The simplest way to do this is with the> command, asfollows:

[jchristi@condense-38 ~]$ ./squares.x >results


13


14/53

This creates a file called results, which you can examine with more, less, oremacs, which contains the output of the squares.xprogram. You will use this

results file to generate some graphs.

1.2.4 Real numbers

So far, you have only used integer numbers. Using real numbers is slightly morecomplicated. Later we will discuss the way numbers are stored in the computersmemory. For the time being, lets assume that you want to use real numberswith the highest accuracy, so-called double precision. (Single precision realnumbers also exist; they take up less memory, but are less accurate, and wewill not use them in this course.) Each compiler implements double-precisiondifferently. Look at the following code:

program real

implicit none

integer, parameter :: dp=kind(1.0d0)

real(kind=dp) i

integer j

i=2.7391d0

do j=1,10

write(*,*)j*i

enddo

end program

Firstly, all real numbers are written in the form 1.234d5 for 1.234 105. Thed tells the compiler that this number must be double precision. In the firstline of code, the parameter dp is set to be equal to the kind of1.0d0. Thismeans that the compiler will do the job of finding out the exact specificationof double-precision numbers for you. All you have to do is set thekind of anyremaining real numbers to dp, as done in the next line to the real number i.

The other new feature in this code is that dp is declared as a parameter. Thisis FORTRANs way of declaring a value that is not going to change during thecourse of the program. Any variable can be declared as a parameter, and thevalue must be set at the same time as the declaration. This has two benefits:if you know a variable should not change during the program, but accidentallydo change it, the compiler will give you an error, and secondly, it allows thecompiler to make the code faster.

14


15/53

1.2.5 if statements and logical constructs

Many times, you will want to write code that is only executed if certain otherconditions are fulfilled. For this task, you can use anifstatement. For example,here is a code snippet2 which computes the square root of a number x, but onlyif that number is greater than zero.

if(x.gt.0.0d0) then

y=sqrt(x)

write(*,*)y

else

write(*,*)y is not greater than zero

endif

The first thing an if statement tests is to see whether the statement in thebrackets, in this case, x.gt.0.0d0, is true or not. If it is true, then the first setof commands is executed, if it is not true, then the second set of commands isexecuted. This code also introduces the .gt. operator, which means greaterthan. Similarly, the operator .lt. means less than.

It is possible to have more than two options. Each statement is evaluated inturn, but once one has been found to be true, the program will pass to the endof the if statement.

if(x.gt.1000.0d0)then

write(*,*)x is very big

else if(x.gt.100.0d0)then

write(*,*)x is quite bigelse if(x.gt.10.0d0)then

write(*,*)x is quite small

else

write(*,*)x is very small

endif

Sometimes, you will want to test more than one statement, and execute differentcommands depending on how many of them are true. This can be accomplishedwith.and. and.or. relations. For example, this code multiplies x and y onlyif both of them are positive.

if((x.gt.0.0d0).and.(y.gt.0.0d0))then

z=x*yendif

2From now on, I will often write only the relevant pieces of code. Everything I write will

be easily transferable into legitimate FORTRAN code by simply declaring the variables, and

putting the start and end program commands.

15


16/53

This code multiples x and y if either or both of them are positive.

if((x.gt.0.0d0).or.(y.gt.0.0d0))then

z=x*y

endif

Here, I have also written anifstatement without an accompanyingelseclause,which is acceptable.

Numbers are not the only type of variables, another type are logical variables,which can take the values of either true or false. In the following code, a logicalvariable called sunny is first declared. The program then sets it either true orfalse, and executes the relevant subsection of code.

logical sunny

! sunny set here

if(sunny)then

! go outside

else

! stay inside

endif

The .not. construction can be used to write an equivalent code. Now the ifstatement is only executed if the logical variable sunnyis not true.

logical sunny

! sunny set here

if(.not.sunny)then

! stay inside

else

! go outside

endif

I will also alert you to a very common mistake. I have made this one manytimes, and I expect you will too. Lets imagine you want a piece of code to beexecuted ifx is equal to 2. The correct form is

if(x.eq.2)then

! execute this code

endif

16


17/53

The.eq. relation is a test which compares whether the two variables are equal.It should always be used for this. The mistake is to write if(x=2). This will

set x equal to 2, which could easily cause mistakes in the rest of the code, andwill cause the if statement to always evaluate as true.

While on the sub ject, another common mistake is not initialising variables. Takea look at the following code, which multiplies two integers i and j, if they areboth positive.

program if

implicit none

! This program can cause errors in certain cases

! BE CAREFUL!

integer i,j,k

i=3

j=-2

if((i.gt.0).and.(j.gt.0))then

k=i*j

endif

write(*,*)k

end program

If i and j are both positive, there is no problem, but if not, as in the aboveexample, then the program has no value ofk. It will output to the screen, orcarry on to the rest of the program, some random piece of memory which it hasread. A good compiler should be able to catch these mistakes for you, but youshould be very careful. I once lost a lot of time due to an error caused exactlythis way.

1.2.6 Mathematical functions

FORTRAN has many inbuilt mathematical functions, such as sin, cos, tan,sqrt, exp and log. Inverse trigonometric functions are given by asin, acosand atan. Some of them are used in the code below:

program mathexamples

implicit none


17


18/53

real(kind=dp) a,b,c,d,e

real(kind=dp), parameter :: pi = 4.0d0*atan(1.0d0)write(*,*)pi = ,pi

a = sin(pi)

b = sin(2.0d0*pi)

c = cos(2.0d0*pi)

d = exp(pi)

e = log(d)

write(*,*)a,b,c,d,e

end program

Note that the answers are not exactly right. sin() and sin(2) should both be

0, for example, and they are very slightly different, whilst the value of itselfis also slightly wrong. Some causes for this behaviour will be discussed later.

1.2.7 Matrices

Very often in science, we will need to deal with matrices, often called arrays inFORTRAN. Fortunately, these are easy to use. In FORTRAN, an array mustcontain all elements of the same type, that is, all integers, or all reals, or alllogical variables etc.

Declaring and using an array is very simple. In this program, we declare andinitialise a 10 10 identity matrix, with ones on the diagonal, and zeroes else-where.

program array

implicit none

integer identity_matrix(10,10)

integer i,j

do i=1,10

identity_matrix(i,i) = 1

do j=1,10

if(i.ne.j)then

identity_matrix(i,j) = 0

endif

enddo

enddo

end program

18


19/53

In FORTRAN, the first index refers to the column, and the second index refersto the row. For instance, b(5,3) is the element in the fifth column and third

row of the matrix b. If you have learnt some C or C++, you will know thatthese languages refer to the elements the other way round.

Also, in this program, I have put one do loop inside another; this is callednesting loops. I have also used the not equal to (.ne.) relationship.

Something to think about now: write a program which will multiply togethertwo square matrices. You will have to declare the matrices, give some values tothe elements, either by assigning them inside the program or reading them in,and then deduce the right commands to multiply them correctly. I will give acorrect solution later.

1.2.8 Input and output: reading and writing data

You have already seen how to write to the screen. It is often helpful to be ableto read from the screen as well. For example, take a simple yes/no question likethis.

program capitals

implicit none

character answer

write(*,*)Do you want all capitals? y/n

read(*,*)answer

if(answer.eq.y)thenwrite(*,*)HELLO WORLD!

else if(answer.eq.n)then

write(*,*)hello world!

else

write(*,*)You should have typed y or n

endif

end program

This program will wait for you to give an input which it will give to the variableanswer. If you type y, it will print the message in capitals, if you type n, it willuse small letters. If you use neither, you will get a message telling you whatyou should have used.

This program also introduces a new variable type, that ofcharacter. If youwant to declare variables of type character with a given length, they shouldbe declared as follows, for a variable called surnamewith up to 20 characters:

character(20) surname

19


20/53

Rather than reading or writing to the screen, you will often want to read orwrite to files. This is an example from my own work: reading in the positions

of some (72, in this case) atoms:

open(10,FILE=positions)

do i=1,72

do j=1,3

read(10,*)pos(j,i)

enddo

enddo

close(10)

Before reading from a file, it needs to be opened, which is done in the first lineabove. The number 10 here is called the unit number, which is a unique number

between 9 and 99, assigned by the programmer to a specific file. Then, insidetwodo loops, the atomic positions are read in to the array pos. After havingread all the necessary data, the file is closed. Note that if I want to read orwrite from a different file, I must use a different unit number.

The syntax for writing is very similar.

open(11,FILE=newpositions)

do i=1,72

do j=1,3

write(11,*)newpos(j,i)

enddo

enddo

close(11)

These are the very simplest methods of reading and writing to files. Othertechnicalities exist, but you do not need to know them at the beginning.

1.3 Introduction to gnuplot

gnuplotis a popular graph-plotting program. It is quite powerful and has manyfeatures, although I will only touch on the basics here. It should be installed onall the ICTP computers.

gnuplotcan be activated straightforwardly from the command line:

[jchristi@condense-38 ~]$ gnuplot

The copyright information will be displayed, and then at the end, the commandline will read

gnuplot>

20


21/53

Lets look at the results file you created earlier from the squaresprogram. Typeas follows:

gnuplot> plot "results"

A graph should be displayed, with all your data points marked with red crosses.If you prefer a nice clean line, type

gnuplot> plot "results" with lines

which will join the points for you.

You might want to check whether our FORTRAN program has given the correctresults. gnuplotcan plot functions. Type

gnuplot> plot "results" with points, x*x with lines

You will see that the points from our file resultssit perfectly on the line plottedfor us by gnuplot, verifying your program.

Go back to the squares program, and reprogram it to output the cubes as well,by changing the relevant line to

write(*,*)i,i*i,i*i*i

Look at the results file so you know its format.

Start gnuplot again. You can plot the first column (x) against the second column(x2) as before

gnuplot> plot "results" using 1:2 with lines

Now, you can also plot the first column (x) against the third (x3) using


It is also possible to plot x2 againstx3, and Im sure you can figure out how.

It is also possible to get a three-dimensional plot. To do this one must use thecommandsplot.

gnuplot> splot "results" using 1:2:3 with lines

Saving the graphs in postscript format is possible, although the syntax is slightlycomplicated.

21


22/53

gnuplot> set terminal postscript

gnuplot> set output "results.ps"


Then, we have a nice graph in postscript format. Type gv results.pson theLinux command line to have a look. gv is a program to read postscript andPDF files.

gnuplotalso allows you to abbreviate commands to save time. For instance,instead of with lines, you can type w l, and instead ofusing, you can typeu. For example,

gnuplot> plot "results" u 1:2 w l

You can also add titles and axis labels to our graphs, with the set command.

gnuplot> set title Our results

gnuplot> set xlabel x

gnuplot> set ylabel x^2

gnuplot> plot "results" u 1:2 w l

If you want to restrict your plot to a specific range, this can be done usingsquare brackets. For example, to have an x-range from 0 to 10, and a y -rangefrom 0 to 100, we can use:

gnuplot> plot [0:10][0:100] "results" u 1:2 w l

gnuplotoffers many more features. Check outhttp://t16web.lanl.gov/Kawano/gnuplot/intro/index-e.html for a goodonline resource.

1.4 Representation of numbers

There is no unique way to represent numbers in the memory of a computer.While standards exist, and many compilers obey them, these standards havelimitations, and it is important that you are aware of them.

1.4.1 Integers

A computer can only give a finite amount of memory to each number. Thismeans that only a limited set of numbers can be stored. In the case of integers,if 32 bits are used, which is common, the range of numbers available runs from2147483648 to 2147483647. At first, you might think that this would not be a

22


23/53

problem: after all, these are large numbers. Lets take a simple example wherethis lack of storage becomes a problem.

Say I want to sum the numbers from 1 to n. A simple code to do this is asfollows:

program integercrash

implicit none

integer i, total, n

write(*,*)n=

read(*,*)n

total=0

do i=1,ntotal = total + i

enddo

write(*,*)n,total

end program

Make sure you understand what this program does, compile and run it. Itshould output the end of the range and the sum. Try it for some differentranges less than 50000. Now changen to 70000 or some larger number. Youshould see immediately that the program is not giving the right answer. Theright answer, which you can compute easily from n(n+ 1)/2, is too large to be

correctly represented.

1.4.2 Floating-point numbers

For real (non-integer) numbers, the problem is even more acute. In additionto the memory problem that exists for integers, it is not possible to representalmost all real numbers exactly. The usual way of representing them is to use64 bits for each number: 1 for the sign, 11 for the exponent, and 52 for themantissa.

This method of storage is rather complicated, but imagine the number 1.13108.We dont need to write 113000000 to store the number, we just need to store1.13 and 8, if we assume that the computer always uses base 10. Although areal computer uses base 2, it stores numbers in exactly the same way. With 64bits for each real number, this is equivalent to storing 15 or 16 significant figuresin base 10.

This representation brings with it a number of issues and problems that youshould know.

23


24/53

Many real numbers map to the same floating-point number. The computer hasonly a limited range of precision, and so all real numbers closest to a given

floating-point number map to the same one.

Floating-point numbers do not form a group. That is, if you add two floating-point numbers, you will not get another floating-point number. The computerwill return the floating-point number closest to the sum. This will usually notbe exact.

If you add a very large number to a very small one, the very small number islikely not to show up. For instance, adding 1 to 1020 will give 1020. Thereis not enough memory to represent 20 significant figures of a number, and thecomputer will drop all but the first 15 or so.

Subtracting two large numbers which are close to each other will often givea number which is quite wrong. If this number is then used later on in the

program, wrong answers can result.This program illustrates these two problems. The (g25.15) in the writestatement is there to give the output format of the number.

program fperrors

implicit none


real(kind=dp) x,y,z

integer i,j

x = 1.0d20

y = 1.0d0write(*,(g25.15)) x+y

x = 1.0d16+1.0d0

y = 1.0d16-1.0d0

write(*,*)The answer should be 2, but is ,x-y

end program

A simple program shows the range of available real numbers, and what happensonce those ranges are exceeded.

program bigandsmall

implicit none


real(kind=dp) x,y

integer i

24


25/53

x=1.0d0

y=1.0d0

do i=1,250

x = 50.0d0*x

y = y/50.0d0

write(*,*)x,y

enddo

end program

2 Finding zeroes of a function

2.1 Introduction

Consider a real function f(x) of a real variablex. There is known to be a pointx at which f(x) = 0. How do we find the value x?

In this subsection of the course, methods to find the zeroes of a function will bediscussed. For simple functions, these can be found analytically, of course, butfor many more complicated functions, they cannot, and a numerical estimationis required. We restrict ourselves to functions of a single variable. For a generalfunction, the problem is not a trivial one.

Four methods will be shown. They have many similarities, but enough differ-ences that it is worth spending a little time on each to see how they work.

In addition, you will also learn how to use functions in FORTRAN, which willbe of general use.

2.2 bisection

If we know that the zero x lies in a certain range a < x < b, then we can usethe bisection method, which is the simplest of the four methods.

If the position x of the zero is such that a < x < b, then we know thatf(a)f(b)< 0. We simply split the range at the midpoint m = (a+b)/2 and thenfind which half of the range the zero is in now, by testing which off(a)f(m) orf(b)f(m) is negative. We now have a range exactly half the size of the previousrange, and we know that it contains a zero. Figure 1 shows this process (the zerois seen to be in the right-hand half of the range at this step). We then repeatthe process with the new smaller range. After a certain number of times, thismethod will converge onto the zero of the function, within machine accuracy.

The advantage of this method is that it will always find the zero, and it is easyto work out how many steps it will take to do so. (The size of the range will

25


26/53

Figure 1: The bisection method. m= (a+b)/2.

be halved at each step, therefore aftern steps, the size of the range will be 2n

times the size of the original range. When this is equal to the desired tolerance,the zero will have been found.)

The disadvantages of this method are that it is the slowest to converge of the

four methods which we discuss, and that a range containing the zero of interestis needed, as a starting point.

The conditionf(a)f(b)< 0 is not a proof to show that there is a zero located between

aand b. There may, for instance, be any odd number (3,5,...) of zeroes between a and

b. The function might not be continuous between a and b. Also, if there are an even

number of zeroes between a and b, then f(a)f(b) > 0. For these reasons, it is often

very helpful to have a good idea of the behaviour of your function, for example, by

plotting it first. This is a particular case of the general maxim that you must always

think before running a computer program; if you put garbage in, you will only get

garbage out.

Below is a copy of a working program which will find the zero of a function,if it is given the function, a range which contains a zero, and a tolerance levelto tell the program when it is close enough to zero. Recall from the discussionof floating-point numbers that the zero will almost certainly never be foundexactly.

The function is defined in this program separately as an example. In this case, itis probably not necessary because the function is simple. In general, it can take

26


27/53

a lot of work to evaluate a function, and it will usually be clearer to separatesuch an evaluation from the main flow of code.

! This program finds the zeroes of a function

! by bisection, assuming that the zero of the

! function is known to lie in a certain range

program zeroes

implicit none


! f is the function

! The zero lies between a and b

! m is the midpoint

real(kind=dp) f

real(kind=dp) a,b,m

! the program will stop when it gets to within

! tol of zero

real(kind=dp), parameter :: tol = 1.0d-10

! Set range

a = 1.0d0

b = 2.0d0

! Check that the zero lies between a and b

if(f(a)*f(b)>0.0d0)thenwrite(*,*)ERROR! No root between a and b

endif

! This loop is executed as long as the absolute value

! of f(m) is less than tol

do while (abs(f(m))>tol)

m=(a+b)/2.0d0

if(f(a)*f(m)


28/53

! function definition

! I cannot declare a variable dp outside a function, so I! have set the kind of these functions to the kind of 1.0d0

! directly

real(kind=kind(1.0d0)) function f(x)

real(kind=kind(1.0d0)) x

f = x * x - 3

return

end function

The function is defined after the program ends. Note that it is also required tohave a type (e.g. real, integer). The first line must declare the name of thefunction, in this case f, and the arguments that the function takes, in this case

only one, called x. xalso needs to be declared. The definition of the function inthis example is f=x2 3; you can of course change this. The keyword returnis then used to return the value of f(x) to the main program. The functionmust end with end function.

In the main program, I have also used a do while loop. Previously, you haveseen a do loop which is executed a certain, specific number of times. In thiscase, the doloop is executed only while a certain condition is true.

After compiling and running the program, you will see that the program outputsan approximation to x = +

3. Starting from a range including the root x =

3 will give that answer. You can experiment by changing the range, or thefunction.

2.3 The regula falsi method

The regula falsimethod (also called the method of false position) is a slightlymore sophisticated method for finding the zero of a function. As before, we willassume that there is only one pointx at whichf(x) = 0, and that it is knownto lie in a certain range a < x < b.

The regula falsimethod approximates the function fas a straight line betweenthe two points at the end of the range. This straight line will cross thex-axisat a certain point m, which forms the next approximation to the zero. Definingthe two points at the end of the range as (a, f(a)) and (b, f(b)), the crossingpointm occurs at

m= b b

a

f(b) f(a) f(b) (1)As in the bisection method, the program must then test to see in which sub-section of the range, a < x < m or m < x < b, the zero occurs. The processis then repeated on the smaller range, using (m, f(m)) and one of the othervalues, such that the new range contains the zero. Figure 2 shows this method.

28


29/53

-5

-4

-3

-2

-1

0

1

2

3

4

0 0.5 1 1.5 2 2.5

Regula falsi method

(a,f(a))

(b,f(b))

(m,0)

Figure 2: The regula falsimethod. The solid line is the function f. The dottedline is the straight-line approximation.

This method can converge faster than the bisection method. A disadvantage isthat an initial range known to contain the zero is needed. Also, the number ofiterations needed to find a solution is not known in advance. It can be usefulto stop the search after a given number of iterations, if the zero has not beenfound.

The code is not included because it is very similar to the bisection method code.The only difference is the method by which m is chosen.

2.4 The secant method

The secant method is very similar to the regula falsimethod, but it does notrequire an initial range which contains a zero. Instead it starts from any twopoints, which we define as (a, f(a)) and (b, f(b)), and calculates where a straightline passing through these two points will cross the x-axis. This point c occursat

c= b b af(b) f(a) f(b) (2)

The points (b, f(b)) and (c, f(c)) are now used in the equation above to forma fourth point, and so on, until the method eventually converges on the zero.

Figure 3 shows this method.

The advantages of this method are that it can be faster to converge than theregula falsimethod, and does not need a range containing a zero. However, ithas the disadvantage that, if the starting points are far away from the initialzero, the method may not converge.

29


30/53

-5

-4

-3

-2

-1

0

1

2

3

4

0 0.5 1 1.5 2 2.5

Secant method

(a,f(a))

(b,f(b))

(c,0)

Figure 3: The secant method. The solid line is the function f. The dotted lineis the straight-line approximation.

The code is not included due to its similarity with the bisection and regula falsimethods.

2.5 Newtons method

Newtons method (also known as the Newton-Raphson method for this case)is the most sophisticated of the four methods we will discuss. It starts from asingle point (a, f(a)) and makes a straight-line approximation to the functionfwhich is tangent to fat the point (a, f(a)). In order to do this, the programneeds to know the derivative of the functionf. In the case of a simple function,this can be provided analytically, or it can be computed by the program.

Defining the derivative offat the pointa as f(a), then the new approximationto the zero is given by b = a f(a)/f(a). The process is then repeated usingthe point (b, f(b)) as the starting point, until the method is sufficiently close tothe zero.

A working code is given below.

! This program finds the zeroes of a function

! by Newtons method

program zeroes_newton

implicit none

30


31/53


! f is the function! df is its derivative

! a is the point

real(kind=dp) f,df

real(kind=dp) a

! the program will stop when it gets to within

! tol of zero

real(kind=dp), parameter :: tol = 1.0d-10

! Set initial point

a = 1.0d0

! This loop is executed as long as the absolute value

! of f(a) is less than tol

do while (abs(f(a))>tol)

a = a-f(a)/df(a)

enddo

write(*,*)An approximation to the zero is ,a

end program

! function definitions

! I cannot declare a variable dp outside a function, so I

! have set the kind of these functions to the kind of 1.0d0! directly



f = x * x - 3

return

end function

real(kind=kind(1.0d0)) function df(x)


df = 2*x

return

end function

Newtons method is the fastest of these four methods to converge. Convergenceis not guaranteed, however. If the derivative is very small or zero, then themethod will fail. Depending on the shape of the function, it is also possible toobtain cyclic behaviour, in which the program returns to points which it hasfound before, and thus becomes stuck in an infinite loop.

31


32/53

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

0.8 1 1.2 1.4 1.6 1.8 2

Newtons method

(a,f(a))

(b,0)

Figure 4: Newtons method. The solid line is the functionf. The dotted line isthe straight-line approximation.

It is possible to define more sophisticated methods. Newtons method computes a

linear approximation to f using its first derivative. Cauchys method computes a

quadratic approximation to fusing its first and second derivatives, and three points.

2.6 Iterative procedures

Newtons method is one example of the solution to an equation found by iter-ating a function. I will give a further example3.

Letp >1. What is the value of the following infinite expression?

a= 1

p+ 1p+ 1

p+...

(3)

We can see that a solution to the above equation can be found iteratively asfollows.

x0 = 1 (4)

x1 = 1

p+ 1=

1

p+x0(5)

x2= 1p+ 1

p+1

= 1p+x1

(6)

... (7)

3from the notes of M. Sellitto and T. Galla.

32


33/53

xk+1 = 1

p+xk(8)

and can be reduced to the following functional iteration

xk+1 = F(xk) (9)

whereFis defined as

F(x) = 1

p+x (10)

One can then compute a to arbitrary precision simply by iterating the abovefunctional form, using a code very similar to that given for Newtons methodabove. Of course, in this case, the answer can be determined analytically:

x = 1

p+x (11)

x2 +px 1 = 0 (12)x = (p

p2 + 4)/2 (13)

3 Linear algebra

3.1 Introduction

In this subsection, we will look at matrices and learn how to solve systems oflinear equations, taking careful note of the problems that can arise.

3.2 Elementary operations

Addition and subtraction of matrices can be carried out straightforwardly inFORTRAN, but multiplication cannot be done in the same way. If you askfor the product of two matrices with a*b, you will get the wrong answer. Theprogram instead creates a matrix in which each element is the product of the el-ements in the same places in the original two matrices. Fortunately, FORTRANprovides us with a function matmul which multiplies matrices correctly.

program elementary_matrices

implicit none

integer i,j

! set the dimension of the matrices

integer, parameter :: ndim = 2

! declare three matrices: a, b and c

33


34/53

integer a(ndim,ndim), b(ndim,ndim), c(ndim,ndim)

! put some values into a and bdo i=1,ndim

do j=1,ndim

a(j,i) = 3*j-4*i

b(j,i) = -1*j+4*i

enddo

enddo

write(*,*)a=,a

write(*,*)b=,b

c = a + b

write(*,*)c=,c

c = a - b

write(*,*)c=,c

! NB. This is not the actual matrix product

c = a * b

write(*,*)c=,c

! The correct way is to use the matmul function

c = matmul(a,b)

write(*,*)c=,c

end program

3.3 Solving systems of linear equations

In this subsection, we will learn how to solve systems of linear equations. As anexample, lets look at the system

2x1 3x2+x3 = 18 (14)4x1+x2 3x3 = 10 (15)x1 4x2+ 2x3 = 20. (16)

We want to find the solution{x1, x2, x3}.

3.3.1 Elementary operations that do not change the solution

We can first ask which operations do not change the solution {x1, x2, x3}. Thereare three of relevance: multiplying any of the equations by a constant value, re-ordering the equations, and adding any multiple of one equation to any multiple

34


35/53

of any other. I give these three without proof, but it is easy to satisfy yourselfthat they will not alter the values xi. This means that we can rearrange our

system of equations using these three operations, for our convenience.

3.3.2 Gaussian elimination

The method below, which solves a set of linear equations, is called Gaussianelimination. It uses the fact that we can add a multiple of one equation to anyother equation without altering the solution, to get rid of some of the coefficients.

We first add -2 times equation 14 to equation 15, to give the equations as follows:

2x1 3x2+x3 = 18 (17)

7x2

5x3 =

46 (18)

x1 4x2+ 2x3 = 20 (19)The value of -2 was chosen specifically to make the coefficient ofx1 zero in thesecond equation.

Then we add 1/2 times equation 17 to equation 19. This gives us

2x1 3x2+x3 = 18 (20)

7x2 5x3 = 46 (21)

112

x2+5

2x3 = 29 (22)

The value of 1/2 was chosen to make the coefficient ofx1 in the third equationzero.

Now, we wish to make the coefficient ofx2 in the same equation zero, which wecan do by adding 11/14 times equation 21 to equation 22. This gives us

2x1 3x2+x3 = 18 (23)

7x2 5x3 = 46 (24)2014

x3 =100

14 (25)

At this point, we essentially have the full solution. It is easy to see from equation25 that x3 = 5. Substituting this value into 24 gives x2 =3. The values ofx3 and x2 can then be substituted into equation 23 to give x1 = 2. The fullsolution is{2,3, 5}.

35


36/53

3.3.3 Computational implementation

The original system of equations 14 - 16 can be written in the form of a matrixequationAx = b.

2 3 14 1 31 4 2

x1x2

x3

=

1810

20

(26)

After the transformations carried out above, equations 23 - 25 can be writtenas

2 3 10 7 50 0 20/14

x1x2

x3

=

1846100/14

(27)

In the following implementation, the matrices are stored as a, x and b respec-tively.

! This program solves a set of linear equations by Gaussian elimination

program gausselimination

implicit none


integer, parameter :: ndim=3 ! dimension of matrix

real(kind=dp) p

integer i,j,k

! a is the set of coefficients, b is the set of answers

! x is the solutionreal(kind=dp) a(ndim,ndim),b(ndim),x(ndim)

! set matrix

open(11,file=matrix.in)

do i=1,ndim

read(11,*)a(:,i)

enddo

read(11,*)

read(11,*)b(:)

close(11)

do i=1,ndim

do j=i+1,ndim! set pivot

p = a(i,j)/a(i,i)

! work out the changed elements

do k=1,ndim

36


37/53

a(k,j) = a(k,j) - p*a(k,i)

enddo

b(j) = b(j) - p*b(i)

enddo

enddo

! Now back substitute to find the answers

do i=ndim,1,-1

x(i) = b(i)

do j=i+1,ndim

x(i) = x(i) - a(j,i)*x(j)

enddo

x(i) = x(i)/a(i,i)

enddo

! and write out the solutions

write(*,*)x

end program

The original coefficients are stored in the file matrix.inwhich has the followingform (note especially the blank line):

2 - 3 1

4 1 - 3

-1 -4 2

18 -10 20

When compiled and run, we see that the correct answers are given.

3.3.4 Pivoting

Although the Gaussian elimination method as described above looks robust, infact it must be handled with care. In particular, it fails on some systems whichare trivial to solve analytically. For example,

0 12 1

x1x2

=

12

(28)

There is no number by which the first row may be multiplied that the coefficientof x1 in the second row will vanish, and the algorithm given above will fail,

37


38/53

despite the fact that the system is solvable by inspection with the answer x=

{1.5, 1

}.

The same difficulty is seen in the following system, although at first it is noteasily evident.

12 1

x1x2

=

12

(29)

If we carry out the Gaussian elimination analytically, we find

10 1 2

x1x2

=

12 2

(30)

This means that

x2 = 2 2

1

2

(31)

We have made no approximations so far. However, the computer may. If issufficiently small, then the 22/and 12/terms will both be evaluated as2/. This means that x2 will be set equal to 1. When we substitute this valueofx2 into the top row, which is

x1+x2 = 1, (32)

this will give us x1 = 0. The true values for very small are{x1 = 1.5, x2 =1 32}, which are consistent with values{1.5, 1} found above for = 0. It isclear that if the computer makes this approximation, the value of x2 will beaccurate, while that ofx1 will be completely wrong.

Fortunately, the problem may be solved straightforwardly. If the order of the

rows are interchanged, such that the equations are now written as 2 1 1

x2x1

=

21

, (33)

and then the Gaussian elimination is performed, the following matrices areobtained

2 10 1 +/2

x2x1

=

21

(34)

Starting from the bottom row, we find that

x1 = 1 1 +/2

, (35)

which the computer will set equal to 1 if is sufficiently small, which, whensubstituted into the top row, will givex2 = 1.5, as it should. We now have theright answers.

This is a classic example of a small numerical instability causing a very largeerror. We must be careful, whenever we use this elimination technique, to

38


39/53

minimise the risk of this happening. This type of row interchange is known aspartial pivoting.

The problem above occurred because the numberwas much smaller than otherelements in the row. We solved the problem by interchanging the rows, whichmeant the first coefficient of the pivot row was of comparable size to the otherelements. The technique of partial pivoting is the act of interchanging the rowssuch that the row with largest magnitude is on the diagonal at each step. (Onecan also use complete pivoting, which is the interchange of rows and columns.However, the extra complexity usually outweighs any benefit.) In the exampleabove, we interchanged the rows so that 2 was on the diagonal, rather than themuch smaller .

Although we did not use partial pivoting in the code above, in practice, it shouldalways be used with elimination techniques of this type.

3.3.5 Singular matrices and conditioning

Consider the matrix equation 1 22 4

x1x2

=

510

(36)

It is clear that both equations are identical, and that there are an infinite numberof solutions for x1 and x2. Consider the matrix equation

1 22 4

x1x2

=

512

(37)

It is clear that the equations are inconsistent, and that there is no solution forx1

andx2. If we were to attempt Gaussian elimination on either of these matrices,we would see a zero appear on the diagonal. This is a sign that Gaussianelimination cannot proceed, and that that the matrix is singular. A singularmatrix has a determinant of zero, and does not have an inverse.

Another way to tell if the system will not have a unique solution is to assesswhether the rows (or columns) are linearly independent. The condition for linearindependence is that it is not possible to construct a linear combination of therows (or columns) equal to zero (except for the trivial case 0x1+0x2+0x3+ . . .).If the rows (or columns) are linearly independent, then a unique solution willexist.

In the above cases, it is easy to see that the equations have no unique solution.As the systems become larger, it become harder. At first sight, the dependency

in the following system is hard to spot, but it can be found from twice the firstcolumn minus the second column plus the third column being equal to zero. IfGaussian elimination is attempted, it will fail.

2 3 14 2 101 2 4

x1x2

x3

=

128

5

(38)

39


40/53

It is not just singular matrices which can cause problems. Certain matricesare very sensitive to the effects of small errors. Such systems are called ill-

conditioned, and are characterised by very large changes in the answer for verysmall changes in the coefficients. A classic example of an ill-conditioned matrixis called the Hilbert matrix, where the elements are Hij = (1 +i+j)

1. Thematrix is not singular, but is very nearly so, and the computation of the inverseof even a 20 20 Hilbert matrix will be all but impossible to do accurately.

3.3.6 Determinants

The traditional method of computing the determinant of a matrix (repeatedlyexpanding in terms of the minors of the matrix) is very time-consuming foreven moderately large matrices. Using Gaussian elimination is computationallymuch cheaper.

The elementary operations stated above change the determinant as follows:interchanging two rows changes the sign of the determinant, multiplying onerow by a constant multiples the determinant by the same constant, and addinga multiple of one row to another does not change the determinant. Therefore,finding the determinant of a matrix is simply a matter of carrying out Gaussianelimination, taking the product of the diagonal elements, and multiplying byany appropriate factors.

As an example, lets consider the matrix 2 3 14 1 3

1 4 2

(39)

The determinant is 2(212) (3)(83)+1(16+1) = 20. After Gaussianelimination, the matrix is 2 3 10 7 5

0 0 20/14

(40)

No row interchanges or multiplications were done to obtain this matrix, so thedeterminant is 27(20/14) = 20, as before. For larger matrices, calculatingthe determinant via Gaussian elimination offers a considerable saving in time.

3.3.7 Finding the inverse

Although the matrix problem Ax = b has the solution x = A1b, it is usual

to solve for x without finding the inverse because of the complexity involved insuch a calculation. Nevertheless, there are times when the inverse of a matrixmust be found. IfA is not singular, then a set of elementary operations P canbe applied to it which will make it the identity matrix, P A= I. It follows thatthe same transformations Papplied to the identity matrix Iwill be equal toA1.

40


41/53

3.3.8 Special matrices: tridiagonal, sparse etc.

The methods above are valid for matrices in general. If the matrix has a definitestructure, with many elements zero, for example, a tridiagonal or sparse matrix,then other algorithms can be found, which are faster. Some of these fall underthe class of iterative methods.

3.3.9 Iterative methods

In certain cases, iterative methods have advantages over the elimination methodswhich we have already discussed. In particular, if the matrix is sparse (mostlyzeros), then they are faster. They also have very small memory requirements,and they are very easy to parallelise (spreading the computational load overmany processors).

The first step is to solve each of the equations for one of the variables. If possible,solve for the variable with the largest coefficient.

Then, we start with an initial estimate of our solution{x1, x2, x3}. This doesnot have to be particularly accurate. We substitute our approximation into theright-hand side of the equations. Evaluating the equations gives us new valuesfor{x1, x2, x3}. These new values, we hope, are closer than the old values.They are then substituted in the right-hand sides of the above equations, andthe procedure is repeated until the difference between successive iterations is

Note that the method can be made even faster by using the updated values ofthe components ofx in the same iteration. This is known as the Gauss-Seidelmethod. This method will converge more quickly, but will take more time if

parallelised.Another profound advantage of these methods is that they are self-correcting,at least to some extent. Any errors which are made (due to rounding or otherproblems) will be removed at successive steps of iteration. A disadvantage isthat the method will not always converge on the solution.

4 Numerical integration

4.1 Introduction

A commonly encountered problem is that of integrating a function. Althoughthere are some programs, e.g., Mathematica, which can integrate a functionsymbolically, there are of course many functions without a closed-form integral.In this subsection we will learn how to integrate functions numerically. Thiswill also require us to look briefly at the problem of function interpolation.

41


42/53

0.5 1 1.5 2 2.5

0

2

4

6

8

10

12

14

16Function to be integrated

Figure 5: An illustration of the trapezium rule. The function to be integratedis approximated by a series of trapezia.

4.2 Trapezium rule

The simplest numerical scheme for integrating a function is called the trapeziumrule. The area under the curve of the function is divided into thin strips of widthx. The curve of the function is approximated as a straight line, such that eachstrip takes the form of a trapezium, hence the name. The area under the curveis then just the sum of the areas of these trapezia. Figure 5 illustrates thisprocedure.

ba

f(x)dx n1i=0

f(a+ix) +f(a+ (i+ 1)x)

2x , (41)

where the integral from a to b is split into n trapezia of equal width x, suchthatx= (b a)/n.From the definition of integration, as x tends to 0, this will give the exactnumerical value of the integral. In the computer, of course, we cannot use sucha small x, but in general, the approximation will be a good one for small x.

A working code is given below. The function to be integrated, in this casef=x2 3, is given in a separate function.

! evaluates an integral using the trapezium rule

program trapezium_rule

implicit none


42


43/53

integer i

real(kind=dp) lhs,rhs

! function

real(kind=dp) f

! the range of integration

real(kind=dp) a,b

! the number of steps

integer n

! width of one strip, all assumed to be the same width

real(kind=dp) dx

! value of integral

real(kind=dp)answer

! set range

a = 1.0d0

b = 2.0d0

n = 10000 ! number of steps

dx = (b-a)/n ! width of one strip

answer = 0.0d0

do i=0,n-1

lhs = a+(i*dx)

rhs = a+((i+1)*dx)

answer = answer + (f(lhs) + f(rhs))*dx/2.0d0

enddo

write(*,*) answer

end program


real(kind=kind(1.0d0)) xf = x * x - 3

return

end function

The above program integrates the function f = x2 3 between the values of

43


44/53

1 and 2, dividing the range into 10000 strips. You can check analytically thatthe answer should be

2/3 and when the program is run, 10000 strips is good

enough to reproduce this answer to a high level of accuracy. The accuracy willdepend on the number of steps chosen. You must choose this number carefully.If you make it too high, the strip will be too narrow to be evaluated correctly,and/or the integral will take a very long time to evaluate. Too small a numberof strips will not be a good approximation to the function. If practical, theintegration should be repeated with different numbers of strips, so that you canbe sure that your answer is between these two limits.

Of course, more sophisticated expressions can be developed. We will go on todiscuss these, but first we must take a look at function interpolation.

4.3 Function interpolation

Lets imagine that we have a set of data (x1, f(x1)), (x2, f(x2)), . . ., but we dontknow the function f. We can approximate a set ofn+ 1 data points with apolynomial function of order at leastn. I state without proof that there is oneand only one polynomial of degree n which will pass through all the data points.

As a concrete example, take the points (1,2), (3, 12), and (1, 6). Since thereare three data points, I know that I can find a quadratic function which passesthrough all three points.

There is a simple method of doing this. If I define my data points as (xi, f(xi)),then this quadratic P2(x) will fit the data points:

P2(x) = (x x2)(x x3)(x1 x2)(x1 x3)

f(x1) + (x x1)(x x3)(x2 x1)(x2 x3)

f(x2)+ (42)

(x x1)(x x2)(x3 x1)(x3 x2) f(x3) (43)

It is easy to see that in the case where x is equal to one of the xis, all but oneof the terms will vanish because the numerator will be zero. The term whichremains will be equal to f(xi), because the numerator and denominator areequal.

This equation is obviously quadratic in x, and passes through the three pointswe have been given. Some simple algebra shows that (11/4)x2 4x 3/4 is ourinterpolating function. We can then use this function to evaluate intermediatevalues off(x). This procedure is very simple to implement computationally.

Lets write this more generally. If I am interpolating n data points, with apolynomial of order n 1, then the general formula will be

Pn1(x) =n

i=1

nj=1,j=i

x xjxi xj f(xi) (44)

44


45/53

Note that this method will work whether the data are evenly spaced or not, aslong as thex-values are distinct. This type of interpolating polynomial is known

as the Lagrangian polynomial. Of course, one can interpolate with other typesof functions as well, but that is beyond the scope of this course.

4.4 Simpsons rule and more complicated approximations

How does function interpolation relate to evaluating the integral of a function?Recall in the trapezium method that we approximated the function between twopoints as a straight line. Another way of saying this is that we linearly interpo-lated between those two points. In the mathematical language of the subsectionabove, the endpoints of a single strip were x1 and x2, and we constructed afunctionP1(x), which was a linear function between those two points.

The form ofP1(x) is straightforward from equation 44.

P1(x) = x x2x1 x2 f(x1) +

x x1x2 x1 f(x2) (45)

It is clear that this function is linear in x, and has the values f(x1) and f(x2)at the points x1 and x2 respectively.

We have approximated our function f(x) by this linear function P1(x) in theregionx1 < x < x2. If the approximation is a good one then x2

x1

f(x)dx x2x1

P1(x)dx (46)

The integration of the polynomial P1(x) is easy: x2x1

P1(x)dx=

x2x1

x x2x1 x2 f(x1) +

x x1x2 x1 f(x2) (47)

=f(x1)

x2

2(x1 x2) xx2x1 x2

x2x1

(48)

+f(x2)

x2

2(x2 x1) xx1x2 x1

x2x1

(49)

=f(x1) +f(x2)

2(x2 x1) (50)

This is exactly the trapezium rule which was discussed above. The connectionbetween function interpolation and numerical integration should now be clear.The trapezium rule is merely the simplest interpolation that can be made over asmall range with a polynomial function. More complicated expressions, that is,polynomials of higher degree, can be used as approximations in a given range.

45


46/53

We move on, and approximate a range by a quadratic function, that is, wedivide an integral into strips of width x, and approximate the function in

each strip using a function of type P2(x), a quadratic evaluated at points x1,x2 = x1+x/2, and x3 = x1+x.

We use the formula as quoted above:

P2(x) = (x x2)(x x3)(x1 x2)(x1 x3) f(x1) +

(x x1)(x x3)(x2 x1)(x2 x3) f(x2)+ (51)

(x x1)(x x2)(x3 x1)(x3 x2) f(x3) (52)

which will lead, after a lot of algebra, to Simpsons rule:

x3

x1

f(x)dx= (x3

x1)f(x1)

6

+2f(x2)

3

+f(x3)

6 (53)

Simpsons rule is only useful if we can evaluate the function at the intermediatevalue ofx2 = (x1+ x3)/2. If we are trying to integrate over an experimentaldata set, for example, this will not be possible, but the trapezium rule will stillwork.

The procedure for integrating a function with Simpsons rule is exactly thesame as that for the trapezium rule. The range is divided into thin strips, thecontribution of each of which to the integral is evaluated using the relevant ap-proximation. The total value of the integral is just the sum of the contributions.

4.5 Varying strip size

So far, we have only looked at integrating data in equally sized strips. Thereare two cases in which this is not desirable. Firstly, if we are integrating actualdata, they may not be evenly spaced, and secondly, some functions may vary alot faster in some regions than in others. It would be helpful, therefore, to havea method of varying the step size such that it is larger where the function isflatter, and smaller where the function varies more quickly.

Of course, we could just plot the function and vary the step size by hand, perhapsrunning the program separately on a series of ranges with an appropriate numberof steps, and then summing the results to get the total value of the integrand.However, it would be better to have an automated scheme. Schemes where thestrip size varies are often called adaptive quadrature schemes.

The principle of the method is straightforward. The overall integration range issubdivided into strips as before. Each strip is integrated using Simpsons rule.Each strip is then split in half at the midpoint, and the two pieces are eachintegrated using Simpsons rule. If the sum of these two integrands is withina user-specified tolerance of the integral of the full strip, then it is accurate

46


47/53

enough, and the program moves on to the next strip. If not, each sub-strip isdivided in two, and the process is repeated, until the strip is small enough that

the process gives an answer within the tolerance.

The errors will be additive, so if you want the whole integral to be evaluatedwith a maximum error of , and there are n strips, then each one must beevaluated to within a tolerance of less than /n.

4.6 Errors in numerical integration

The errors in these methods can be computed by comparing an expansion ofthe function over the integration range with the approximation. To save youthe bother, I will quote without proof the magnitudes of the error here. For thetrapezoid rule, the error is

112

(x2 x1)3f2(x), (54)

and for Simpsons rule, the error is

190

(x2 x1)5f4(x), (55)

where the endpoints of one strip are assumed to be x1 and x2, and fn(x) is

the nth derivative of f evaluated at x. From these results, it is clear thatSimpsons rule is considerably more accurate. In fact, using a cubic interpolatingpolynomial in each strip is no more accurate.

5 Numerical differentiation

5.1 Introduction

In this subsection, we will discuss techniques for numerically differentiating aset of values. In particular, we will discuss carefully the errors involved in suchtechniques, and ways of minimising them.

5.2 The first guess: the forward-difference approximation

The definition of the derivative f(x) of a function f(x) is

f(x) = limx0

f(x+x) f(x)x

(56)

Of course, in a computer we cannot take the limit x0, so we must inventalternative schemes. Based on the above equation, a reasonable first guess of a

47


48/53

method of differentiating a function is simply

f(x) = f(x+x) f(x)x

, (57)

wherex is as small as possible without compromising on accuracy. Of course, ifwe are taking the derivative of a data set, then x is the separation inx betweenthe data points. If this is too large, then the calculated derivative will not beaccurate at all.

By calculating the Taylor expansion of f(x), we can assess the error in thisapproximation for the derivative. The Taylor expansion off(x) is

f(x+x) = f(x) +xf(x) +x2

2! f(x) +

x3

3! f(x) +. . . , (58)

so rearranging, we find

f(x) = 1

x(f(x+x) f(x)) x

2!f(x) x

2

3! f(x) +. . . , (59)

Becausexis always very small, we can neglect the term in x2 and all smallerterms to say that the error in this approximation is of order x.

5.3 The centred-difference approximation

It would be preferable to have an approximation with a smaller error. Fortu-nately, this can easily be achieved. We can create Taylor expansions based onequation 58 for the functions f(x+x) and f(x x). We can then constructthe difference of the twof(x + x)f(xx), and we can see that every secondterm in the expansion will be cancelled, as follows:

f(x+x) = f(x) +xf(x) +x2

2! f(x) +

x3

3! f(x) +. . . (60)

f(x x) = f(x) xf(x) + x2

2! f(x) x

3

3! f(x) +. . . , (61)

hence the difference is

f(x+x) f(x x) = 2xf(x) + x3

3 f(x) +. . . . (62)

This equation can be arranged to give

f(x) = f(x+x) f(x x)2x

x26

f(x) +. . . . (63)

Now, if we only use the first term in this approximation, we see that the largestterm which we ignore is of order x2, which is significantly smaller than x.This method is much more accurate than our first guess.

48


49/53

A working code for this is given as follows. We will be differentiating the functionf(x) = x4

3 at the point x= 1. It is easy to calculate that the value of the

derivative at this point is 4.

! uses the centred-difference method to differentiate a known function

program diff

implicit none


! The function

real(kind=dp)f

! The step size delta xreal(kind=dp), parameter :: dx = 1.0d-7

! The point at which we wish to differentiate

real(kind=dp), parameter :: x = 1.0d0

! The derivative

real(kind=dp) df

df = ((f(x+dx)-f(x-dx))/(2.0d0*dx))

write(*,*)The derivative is ,df

end program



f = x**4-3

return

end function

On my laptop, with a step size of x = 107 as above, the answer given forthe derivative was 4.000000000115022,4 which is pleasingly accurate. It will beinstructive to investigate the behaviour of the answer as we vary the step size.

When I varied the step size, I obtained the answers given in Table 5.3. Your

answers will be different due to the different step sizes. Recall that the realanswer is exactly 4.

4Your value will very probably be different because you are using different computers.

When doing this for real, you must always test the accuracy of your program on the system

it will be run on.

49


50/53

x Calculated derivative100 8

101 4.04102 4.000400000000015

103 4.000003999999668104 4.000000039998675105 4.000000000403681106 3.9999999998929776107 4.000000000115022108 3.9999999978945757109 4.000000108916878

1010 4.0000003309614831011 4.0000003309614831012 4.0001335577244381013 3.99902333469981341014 3.9968028886505631015 4.218847493575595

1016 2.22044604925031261017 01018 01019 0

Table 1: The value of the calculated derivative and how it depends on x. Theexact answer is 4.

50


51/53

The behaviour is much as we expect. For large x, the calculated value ofthe derivative is quite wrong, but it becomes more accurate as x is reduced,

maximum accuracy for these values being obtained for a step size of 107. Asthe step size grows larger, round-off and computer arithmetic errors becomemore important, and the answer becomes progressively less accurate. As thestep size becomes smaller than about 1012 or so, the errors grow massively,and once again the calculation becomes hopelessly wrong.

Recall though that I said that the error of our expression was of the order ofx2,which is a very small number, far smaller than any of the errors seen above. Thereason is that the errors given in the table are caused by numerical error, theeffects of which far outweigh the effects of the truncation error. For example,we see from the formula forf(x) that any errors in f(x x) are multiplied bythe number 1/2x. Sincex is very small, this fraction is very large and greatlymagnifies any errors in the evaluation of the function.

As ever, this emphasises the need for robust and careful testing of your programsbeforeyou start using them to calculate important results.

5.4 Higher-order derivatives

By keeping more terms in the Taylor expansion, it is possible to construct com-putational expressions for higher-order derivatives, although careful attentionshould be paid to the accuracy of these schemes. It follows that if differentiatinga function introduces certain errors, differentiating it again introduces the sameerrors on the already slightly wrong answers.

The expression for the second derivative which can be found by substituting our

expression for f(x) back into the main Taylor expansion is

f(x) = 1

x2[f(x+x) 2f(x) +f(x x)] (64)

It carries with it an error of the order ofx2.

5.5 Differentiation by function interpolation

Just as the trapezium rule and Simpsons rule from the integration subsectionare connected to function interpolation, so the numerical rules for differentiationcan be recovered in the same way. For example, lets assume we are approxi-mating three pointsx

1, f(x

1),x

2, f(x

2) andx

3, f(x

3) by a quadratic polynomial

exactly as we did in the integration subsection.

Then we have

P2(x) = (x x2)(x x3)(x1 x2)(x1 x3) f(x1) +

(x x1)(x x3)(x2 x1)(x2 x3) f(x2)+ (65)

51


52/53

(x x1)(x x2)(x3

x1)(x3

x2)

f(x3) (66)

Of course, we can straightforwardly take the derivative of this approximation,

P2(x) = (2x x2 x3)(x1 x2)(x1 x3) f(x1) +

(2x x1 x3)(x2 x1)(x2 x3) f(x2)+ (67)

(2x x1 x2)(x3 x1)(x3 x2) f(x3) (68)

If we evaluate this expression at x = x2, we obtain

P2(x2) = (x2 x3)

(x1 x2)(x1 x3) f(x1) + (2x2 x1 x3)(x2 x1)(x2 x3) f(x2)+ (69)

(x2 x1)(x3 x1)(x3 x2) f(x3) (70)

The above equation is general. We can link it to the other work if we assumethat the three points are evenly spaced, and occur at x1 = x x, x2 = x,x3 = x+x, then we find

P2(x) = 1

2x f(x x) + 1

2xf(x+x) (71)

But this is just the centred-difference expression for the derivative we havealready used.

5.6 Richardson extrapolation

We have discussed at some length in this course the errors inherent in computingvarious properties, and the care needed in programming and running to takeaccount of them. In many cases, the sizes of the errors can be reduced bya clever choice of algorithm. Richardson extrapolation is one such techniquewhich allows us, in this case, to calculated formulas for the derivatives witheven greater accuracy.

Above, I have given two formulas for computing f(x). They had errors of theorder ofx andx2 respectively. Using Richardson extrapolation, we can createformulas with more accuracy.

From the Taylor expansions of f(x+ x) and f x, we can see that in thedifference, all the terms with an even power ofx will vanish:

f(x+x) f(x x) = 2xf(x) + 23!

x3f(3)(x) + 2

5!x5f(5)(x) +. . . (72)

52


53/53

which, when rearranged, gives

f(x) = 12x

[f(x+x) f(x x)]

13!

x2f(3)(x) + 15!

x4f(5)(x) + 17!

x6f(7)(x) +. . .

(73)wheref(n)(x) is thenth