,

),lay, 1958

..

FLORIDA STATE UNIVERSITY

ITERATIVE METHODS FOR THE SOLUTION

OF LINEAR EQUA1:IONS

By STANLEY R. pENDER,

A Paper Submitted to the -Graduate Council of Florida State University in partial fulfillment of the re~uirements ' for the degree of Master of Science.

Approved :_·JA~(l~'~IQ~-.J:""":'1..1..c~~~~!4-_ Professor D

none lfinor Professor

~~e~/ R 11

Dean of the Graduate School

FLORIDA STATE UNIVERSITY LlBRARIBB

MAR 14 2001

TALLAHASSEE, FLORIDA

FLORIDA STATE UNIVERSI TY

ITERATIVE ME HODS FOR THE SOLUTION

OF LINEAR EQUATIONS

BySTANLEY R. I ~END~,

A Pa erSubmitted to the Graduate Council ofFlorida State University in partialfulfillment of the requirements 'forthe degre of Master of Science.

Approved: .p~Professor D

Yay, 1958

noneMinor Professor

FLORIDA STATEUNIVERSITY L1BRARIBS

MAR 14 2001

TALLAHASSEE, FLORIDA

4CKNOlILEDGEMENTS

I wish to thank Dr. Paul J. llcCarthy for his patient assistance

in the preparation of this paper.

11

I wish to thank Dr. Paul J. McCarthy for his patient assistance

in the preparation of this paper.

11

TABLE OF CONTEtITS

ACKNOWLEDGEMENTS • • • • • • • • • • • • • • • • Page

11

Chapt.er I. INTRODUCTION • • • • • • • • • • • • • • 1

II. ITERATIVE METHODS

l~ Analytic basis • • • • • • •• •• 7 2~ Method or steepest descent • • • • • 10 3; Gauss-8eidel method •••••••• 12 4; Method of · relaxation • • •• ••• 13 5. Summary • • • • • • • • • • • • • • 14 6. Examples • • • • • • • • • • • • • • 16

BIBLIOGRAPHY • • • • • • • • • • • • • • • • • • 21

iii

TABLE OF CONTmITS

ACKNOWLEOOEMENTS • • · . • • • • • • . . . . . . Pageii

ChapterI. INTRODUCTION • • • • • • • • • • • • • • 1

II. ITEM.TIVE METHODS

l~ !nalytic basis • • • • • • • • • • • 12~ ethod of steepest descent • • • • • 103~ Gauss-Seidel method ••• • • • •• 12.4~ Method ot'relaxation • • • • • • • • l}5. Summary • • • • • • • • • • • • • • ]lJ.6. Examples • • • • • • • • • • • • • • 16

BIBLIOGRAPHY • • • • • • • • • • • • • • • • • • 21

iii

CHAPTER I

INTRODUCTION

The numerical solutions of ma~ types of problems are .generally

obtained by solving approxi'll8ting linear algebraic systems. Moreover,

in solving a nonlinear problem, one may replsce it by a sequence of

linear systems providing pr.ogressively improved approximations. l For

the study of these linear systems of equations a geometric termin-

ology with the compact symbolism of vectors and matrices is useful.

A resume ~f the basic principles of higher algebra necessary for the

development of the material to follow is therefore included.

A vector x of order n is defined to be a set of n real numbers

Xl' %2' ••• , xn arranged in a definite sequence and is written

x • (Xl' %2' ••• , xn). The vector 'f is wr:I. tten

Y - (Yl' Y2' ••• , Yn).

Equali ty of vectors x - y is defined to mean simultaneous equaU tr of

similarly numbered real number elements, xt • Yi for i • 1, 2, ••• , n.

The multiplication of 1by any real number c is defined as the

vector

!Householder, Alston S., P~incip1es of Numerical Analysis (New Yorb McGra_Hill Book Company, Inc., 1953) p. 44.

1

CHAP'TER I

INTRODUCTI ON

The numerical solutions of many types of problem aTe generally

obtained by solving approxl1!Jating linear algebraic systems. Moreover,

in solving a nonlinear problem, one may replace it by a sequence of

linear systems providing pr.ogressively improved approximations.l For

the study of these linear systeJllB of equations a geometric termin-

ology lIi th the compact symbolism of vectors and matrices is useful.

A resume of the basic principles of higher algebra necessary for the

development of the material to fol101'f' is therefore included.

A vector x of order n is defined to be a set of n real numbers

Xl' x2 ' • • • , xn arranged in a definite sequence and is WTitten

x • (Xl' x2' ••• , xn).

The vector y is written

Y• (Y1' Y2' • • • , Yn).

Equality of vectors x • -; is defined to mean simultaneous equality of

similarly numbered real number elements, ~ • Yi for i • 1, 2, ••• , n.

The multiplication of X by any real number c is defined as the

vector

IHouseholder, Alston S., Principles of Numerical Analysis (NewYork: McGraw-Hill Book Company, Inc., 1953) p. ,44.

1

2

Addition of two vectors x + y is defined by

(2) it + y • (xl + YIJ ~ + Y2J ••• , xn + Yn).

The totality of all vectors for a fixed integer n with the operations

(1) and (2) is called an n-dimensional vector space over the field of

real numbers R and is written Vn(R). A 8ubeet of the vectors in the

vector space Vn(R) which is closed with respect to the two basic opera­

tions of vector algebra' (1) and (2) is called a vector subspace of Vn(R).

From (2) it follows that there is a unique null vector

~ • (OI, 02, ••• , On)

wi th the property that for any vector xJ

, for all it in Vn(R) and all c in R. For any vector ~ in Vn(R) there

is a unique vector x, such that x +~, • 0, namely the vector

:t' • (-xl' -~, • • , -Xu).

This vector x' is called the negative otiC and is usually denoted by

4. Since x' • (-l)~J the definition at -it may be written -1 • (-1)%.

Subtraction of vectors is defined by the equation x - '1 • x + (-1)1.

Vectors also have the following properties, valid tor all vectors x, Xl.

~ •••• , ~ in V (R) and all scalars c J cl' c2' ••• , cn in R:

(Xl + x2) + x, • Xl + (x2 + 'x,) ,

Xl + x2 • t2 + Xl,

c(xl + x2 + .••• + Xu) • cXl + c~ + ••• + c~,

{cl + c2 + ••• + cnlx • clx + ciX + ••• + cJ:. and

(clc2)'% • cl (c2'f).

2

Addition of two vectors i + Yis defined by

(2) x + Y• (xl + Y1' ~ + Y2' • • • , xn + Yn).

The totality of all vectors for a fixed integer n w-lth the operations

(1) and (2) is called an n-dimensional vector space .over the field of

real numbers R and is written Vn(R). A subset of the ~c.tors in the

vector space Vn(R) whieh is closed with respect to the two basic opera­

tions of vector algebra (1) and (2) is called a vector subspace of Vn(R).

From (2) it f'ollows that there is a unique null vector

~ • (0}, 02, • • • • on)

'Wi th the property that for any vector ~,

, "0 • 'x • 0 and

f'or all x in Vn(R) and all c in R. For any vector x in Vn{R) there

is a unique vector~' such that x + x' '" 0, namely the vector

x' • (-Xl' -X2, • • , -xn).This vector xt is called the negative of x and is usually denoted by

-x. Since x' • (-l~, the definition of -x may be written -~ • (-1)%.

Subtraction of' vectors 1s defined by the equation x - 'i • x + (-l)"YO.

Vectors also have the following properties, valid for all vectors x, Xl.

x2, ••• , ~ in V (R) and all scalars c, el' c2' ••• , cn in R:

(Xl + x2) + x3 • Xl + (x2 + "X3)'

Xl + 'x2 • :t2 .. Xl'

e(xl + x2 + .•• e + Xn) - eXI + eX2 + • • • + exn,

(01 + c2 + e .• • + cnli • C1X + C2X + • e • + c~ and

(c1c2):X • cl(c2~)·

If any n vectors :tl , ~, •••• ~ lie in a vector space Vn(R).

they are called linearly dependent if there are scalars ch c2. • •

• • cn• not all zero, such that

c 'Ij. + c ~ + ••• + c X • 0. l-~ 22 nn

If no such scalars exist. the set~, x2 •••• , -in is called lin-

early independent. A vector space is .-dimensional if m is the max-

imIm number of linearly independent vectors in the space. Any 11\

linearly independent vectors in the space then constitute a coord i-

nate system for the space and any vector in the space can be written

as a linear combination of these linearly independent vectors.

We shall refer to the following set of n-dimensional vectors of

Vn(R) as the Ii coordinate systeJl1 for Vn(R)

el

• (1, 0, 0, • •• , 0)

"2 . (0, 1, 0, 0 • . . . , 0) • • • • • • en • (0. 0 •• • • , 0, 1) •

With regard to the vector X the scal:lrs let. for i • 1, 2, ••• , n

are called the coordinates of i in the. s;ylItem ar.d the vectors let1!i

are ita components.

When the coordinates of -f in the e fI;ylItem are arrsnged in column

form - the arrangement is known as a column matrix which we shall

deeignate by xe. The arrangeJl1ent (~l' ~ ••••• ~n) is called

a row matrix. Any rectangular array of numbers i4 called a rectan-

3

If any n vectors iI' ~2' ••• , ~ lie in a vector space Vn(R),

they are called linearly dependent if there are scalars clJ c2_ • •

• , cn, not all z ro, such that

c y. + e x +. • • + c'x • o.r1. 22 nn

If no such scalars exist, the eet~, )[2' ••• , .~ is called lin-

early independent. .It.. vector space is m-dimensional if m is the max-

!mum nuD!ber of linearly independent vec'tors in the space. Any m

linearly independent vectors in the space then constitute a cQordi-

nat.~~ system for the space and any vector in the space can be written

as a linear combination of these linearly independent vectors.

We shall refer to the following set of n-dimensional vectors of

Vn(R) as thee coordinate system for Vn(R)

e1 • (1, 0, 0, • • • , 0)

....(0, 1, 0, 0, 0)e2 • • • • ,

• •• •• •en .. (0, 0, • • • , 0, 1).

With regard to the vector x the scab.rs %t. lor i • 1, 2, ••• , n

are called the coordiDCltea of i in th a system ar.d the vectors ~lEi

are its coq:> nents.

When the coordinates of'x in the e system are arranged in column

form the arrangement is known as a column matr1Jc which we shall

designate by xe• The arrangement (XJ.1' XJ.2' ••• , "1n) is called

a row matrix. Any rectangular array of numbers is called a rectan-

4

gular matrix:-

'an 812 • • • aln a21 a22 • • • a2n

• • • • • • A • • • • • • • • [aijl~;

• • • • • •

aml 8M2 • • • a mn

throughout the discussion upper case Latin letters will be used to

designate matrices. The double subscripts indicate the tlro-<IIay

ordering of JIIItrices. The fint subSCript is the rOlf index 8nd the

second subscript is the column index.

Given any two matrices A and B • {bijJ~ where i • I, 2, ••• , p

and j ·1, 2, ••• , q; the" product AB is defined as the matrix n

C • r Ci!dm where cn • {aijbjkif n • p. The product III • D· dik R " q J-l

'" lI'here dik • j~' bija jk if III • q. In general JoB of BA. The identity

matrix under multiplication, I • fk:tjS~J is such that its elements

k:tj is equal to one it i • j and is equal to zero it i of j for all

values of i and j. The matrix I is therefore a diagonal matrix; that

is, it is square (has the same number of columns and rows) and all

elements off the main diagonal are zero.

The transpose of the matrix A, designated by attaching a super­

script t to the matrix notation At • {aij'J: is such that for all aij

of A and all aji' of At, aij • a ji ' for all values of i and j. The

transpose of the column ma trix xe is the rOlf matrix xe t • (XIV x12'

• , • , xln)' The transpose of a rOlf matrix is a column matrix,

(Xet)t • xe ' The transpose of

4

gular matrix:-

"all 812 • • • aln

a2l a22 • • • 82n

• • • • • •A • • • • • • • • [aij}~;

• • • • • •aml Que • • • amn

throughout the discussion upper case Latin letters will be used to

designate matrices. The double subscripts indicate the two-way

ordering ot matrices. The fint subscript is the row index and the

second subscript is the column index.

Given any two matrices A and B • {bijJ~ where i • 1, 2, ••• , p

and j • 1, 2, ••• , q; the product AB is defined a8 the matrixn

G • fCitJ: wher cik· ~I 8 ijb jkif n p. The product BA. • D· dik KWI

where dik • j~r bij8 jk ltm • q. In general AB 'I BA. The identity

matrix under multiplication, I • f~j1R, is such that its elements

~j is equal to one if i • j and i9 equal to zero if i rf j for all

values of i and j e The matrix I is therefore a diagonal matrix; that

is, it is square (has the s me number of columns and rows) and all

elements off the main diagonal are zero.

The transpo.se of the matrix A, designated by attaching a super­

8Cript t to the matrix notation At. { 1jtJ ~ is such that for all aij

of A and all 8ji t of At, 8ij • a ji' tor all values of' land j. The

transpose of the column JIB trix xe is the row matrix xet • (XII' %12'. .The transpose of a r~ trix is a column matrix,

5

m'" 1\1 AB • 2 ~ :::: a b • ~ i c • C is

j~I ~ . ,j" ij jk i'l k' ik 1(1 ' mn

(AB)t • Ct • ~ ~. cld. • t. ~,~, bkjaji • BtA t.

Let 11 , 12, ••• , 1n be another set ot linearly independent

vectors in this n-dimensional vector space; then each tj is express­

ible as a linear colllbiDation of the 9 i 'S and vice versa. If we con­

sider e am f to be row matrices,

e • (~, e2, ••• , en)

f • (fl , t 2 , ••• , f n>

in which the coordinates ot the ei ' s and t j , s, Wi th regard to the ~

am 1 coordinate systems respectively, are arranged column-Wise.

Then 1r9 can Write in abbreviated form,

e • f'E , r • eF • am

Then rtf • (eF}teF • Ftetl!F • Ftrr • rtF. H • Ht • frt. Since H • ~,

H ill called a symmetric matrix. Relati ng the e and r coordinate

system we have

I • ete • (f'E)tfE • Etrtf'E • EtHE.

If both e and t are unit (the only nonzero element of the vector is

equal to unity) orthogonal coordinate systems then H • I and

I • E~ • E~.

The geometriC vector -! can beeKpressed as the numerical vector

Xf, that is, as an n-tuple ot coordinates Xj' in the tj coordinate

systera. and 1r9 can wr1 te xf • Fxe-

We define the scalar or dot produet of two vectors x and Y in

the r coordinate systea by

5

m , n ~ ~

AB • ~ ~ ~ a b • ~ i c • C is'.:.) 110' j'l ij jk ~'-I "., ik

"I n "\ m n

(AB)t • Ct .~ ~,ck1 • ~t~i~/bkjaji • BtAt.

Let 11

, t2

, ••• , 1n

be another set or linearly independent

vectors in this n-dimensional vector space; then each f j is express­

ible as linear combination of the e1 t s and vice versa. If we con­

sider e and t to be row matrices,

e • (~, e2, •• • , en)

t • (fl' 12 , • •• , t n>in which the coordinates or the ei ' s and r j , s, with regard to the "i

and 1 coordinate sYBtems respectively, re rranged column-wise.

Then we can writ in abbreviated rorm,

e • f'E , t eF , am

H is called a symmetric matrix. Relating the e and t coordinate

systems we have

I • ete • (tE)tn • Etrttp; • EtHE.

If both e and f are unit (the only no ero element of the vector is

equal to unity) orthogonal coordinate systems then H • I and

I • Etrm • E~.The geometric vector -y can be spressed as the numerical vector

Xr, that ie, as an n-tuple of coordinates Xj' in the f j coordinate

systeru, and we can write xr • he-

We define the scalar or dot product of t 0 vectore X and :I in

the t coordinate ystelD by

6

and it is a pure number. If x and yare the same vector then the

scalar product is the square of the length,

x • x • IxrJ2 • XttHxr-

The matrix II is Baid to be positive definite since xttHxt • 0 and

equality can hold only when xf • O. Two vectors x and yare said

to be orthogonal to each other it their scalar product is zero.

The scalar product my be thought of geometrically as the product

of the length of one vector by the projection of the other upon it.

In the e coordinate sY!ltem

x • y • xetIYe • Xetye· l!:liYn· YU%tl· Yetxe • Y .1l'.

Similarly we can prove that the scalar product is ctllDllllltative in the

t coordinate 8Y!1tem.

The determinant of a square matrix A, m • n, is defined as

A • • • 8ni n

all permutatiOns ot 1, 2, • • • , n

where t is the number of transp08itions required to restore i l , ~,

••• , in to their natural ordering 1,2, ••• , .n. The matrix A

is nid to be nonsingular if the value of its determinant ill nODlllero.

If the row and column of a particular element aij ot a square

I118triX A are deleted. what remains is called the (n - l)x(n - 1) sub­

DIIItrix ~j~ The scalar dij • (_l)i + jl"\jl is called the cotactor

of aij in A. The n x n matrix D • f djl.l g is called the adjoint ot A

and is denoted by 'A. The inverse of the matrix A (tor m • n) may

now be defined as A-l • A • tAl

6

and it is a pure number. If x and y 8r~ the same vector then the

scalar product is the square of the length,

x • i • /xrJ 2 • xftHxr "

The matrix Ii is said to be positive definite since %ttHxr • 0 and

equality can hold only when xf • O. Two vectors x and yare said

to be orthogonal to each other if their scalar product is zero.

The scalar product may be thought of geometrically as the product

of the length or one vector by the projection of the othe upon it.

In the e coordinate system

x • Y• xetIYe • :xetye· xliY:tl· Yli~l· yetxe • y • -X.

Similarly we can prove that the scalar product is connnutat.i in the

f coordi te system.

The determinant of a square matrix A, m • n, is defined a8

A • ~ (-l)tau 82i1 2

11 permutations

• •

of 1, 2,

•

• • • , n

where t is the number of transpositiona reqUired to restore 11' ~,

••• , in to tht~ir natural ordering I, 2, ••• I .n. The matrix A

is s3id to be nonsingular if t.hc value of its determinant is nonzero.

If the row nd column of 8 particular element 8ij of a square

matrix A are deleted, what remains is called the (n - 1)x(n - 1) sub­

matrix ~j~ The 8calar dij • (_1)1 + j lMijl is c lIed the cofactor

of 8ij in A. The n x n mat.rix D • {dji) ~ is called the adjoint of A

and is denoted by1. The inverse of the matrix A (for m • n) may

now be defined as A-1. A •IAI

CHAPTER II

ITERATIVE METHODS

1. Anaqt:l.c bash. In solving any equation or systea of equat:l.oll8

there are two poss:l.ble approaches. We may be able to f:l.nd a solution

by means of a direct method which prescr:l.bes only a finite sequence of

operations whose c~letion yields an exact solution. When the nuaber

of unknowns ill great, the algebra:l.c ID9thodll become exceedingly labor:l.­

OUII. The solution of a system Of . this type is then most easily found

by the use of an iterative method. Iterative methodll are ellpecially

valuable when machinell for handling the successive iterations effectively

are available, when a good approximation to the lIolution is known or when

we Wish to obtain solutions of adjusted equations.

In lIolving a set of m equations in m unknowns y - Ax • 0 an iter­

ative method is defined as a rule for operating on an approximate solu-

tion ~ in order to obtain an improved solution ~ + 1; such that the

sequence [xJ so defined has the solution x as its limit.

Since the approximation Xo With which one may begin an i terati ve

method does not need to be ftcloseft to the solution x. We l118y start With

an arbitrary xo, however a good approxilllBtion to the solution will apprec-

iably diminish the number of iterations necessary to achieve a desired

accuracy. If no approximation With which to start the iterative IIIIIthod

is available take Xc • 0 unless the main diagonal of the coefficient

7

CHAPTER II

ITERATIVE :METHODS

1. Analytic basis. In solving any equation or system of equations

there are two possible approaches. We may be able to find a solution

by means of a direct method which prescribes only a finite sequence of

operations whose completion yields an exact solution. When the number

of unknowns is great, the algebraic methods become exceedingly labori­

ous. '!"he solution of a system of this type is then most easily found

by the use of an iterative method. Iterative methods are especi81~y

valuable when chines for handling the successive iterations effeetively

are available, when a good approximation to the solation i known or when

we wish to obtain solutions of adjusted equations.

In solving a set of m equations in m unknowns y - Ax • 0 an iter­

ative method 18 defined as a rule for operating on an approximate solu­

tion ~ in order to obtain an improved solution Xp + 1; such that the

sequence f~ so defined has the solution x as i t8 limit.

Since the approximation X o with which one may begin an i tarative

method does not need to be Mclose" to the solution x. We may start wi th

an arb! trary xo, hO'll'ever a good approximation to the solution will apprec­

iably diminish the number of 1tera tiona neeeesary to achieve a desired

accuracy. If no apprOXimation with which to start the iterative method

is available take Xc • 0 unless the main diagonal of the coefficient

1

g

matrix features relatively large terlllllJ then take xo· '"1/all.

Y2/a22 • • •

A large claaa of iterative methods 18 baaed upon the following

geometric idea: take any vector bo and a sequence of Tectora '\> and

define the sequence [bpJ by bo' ~ • ~ _ 1 - ¢pup where the scalar

¢p is chosen so that bp is orthogonal to~. It t.'le vecto!'8 '\> nfill

out" some m dimensional subspace of the vector space Vn(R) then the

vectors ~ approach as a lim t a vector b which is orthogonal to this

m-space. A possible choice for the Tectors ~ is any set of refer­

ence vectors fi of the • space taken in order and then repeated

u.m + i • f i • where v • 1, 2, • • •• Let f • eF and H • FFt; then

we have

bp • ~ _ 1 - ;PUp. and iii nee ~ is orthogonal to '1>

o • '1> • ~ • Up • ~ - 1 - sfpllp • Up

• ~tHbp _ 1 - sfp~tH~

where I • UptRhp. _ l/UptHUp.

If we let y • Ax repreaent the set of m equations in m unknowns

to be solved and ~ represent any approxilllltion to the solution x,

then we have

rp • y - Ax" • Ax - Axp • A(x - xp) • Asp.

The coordinates of the Tector rp are called the residuals of the

equations. The s,mol rp and equivalently Asp are representations

g

matrix features Nlatively large terlDl!ll then take xo • Yl/all.

Y2/8 22•••

A large class of 1terative methods is based upon the following

geometric idea: take any vector bo and 8 sequence of vectors ~ and

define the sequence l bpJ by bo ' hp • bp _ 1 - ¢pup where the scalar

¢p is chosen so that bp is orthogonal to~. If the vectors ~ ufill

out" some m dimensional subspace of the vector space Vn(R) then the

vectors hp approach as a limit a vector b which is orthogonal to thi

space. A possible choice for the vectors ~ is any set of refer­

ence vectors £i of the space taken in order and then repeated

~ + i • f i , where v • I, 2, • • •• Let f • eF and H • FFt; then

we have

bp • ~ _ I - ¢P'i>'and si nee ~ is orthogonal to '1>

o • Up • hp • Up • hp _ 1 - stpUp • Up

• ~tHbp _ I - ;p~tH~

where ~ • UptHbp _ l/uptHUp.

If we let y • Ax represent the set of m equations in m unknowns

to be solved and ~ represent any approximation to the solution x,

then we have

rp • y - Axp • Ax - Axp • A(x - Xp) • Asp.

The coordinates of the vector r p are called the residuals of the

equation. The symbol rp and equivalently Asp are representations

9

of the d.T1ation of the approximation trom the true solution.

Hence either rp or ep can be taken as bp'

Take the matrb: A to be positive definite. Any sY'lltem of

equations w • ax, B nonsingular, can be transformed by a premul­

tiplication of Bt on both sides into a sY'lltem in which the coeffi­

cient matrix is positive definit~. The solution of

y • Btw • Btax • Ax

as also the solution of w • Bx.

Let rp .~, then

., - Axp • bp • bp _ 1 - tlpUp • y - AXp _ 1 - ¢'pUp, and

~ • ~ _ 1 + "pUpf and

~ • ~ - 1 + -Il-1~.

Since Up is an arbitrary set of vectors lie may substitute '1> • Avp

to obtain

xp • xp _ 1 + t/pvp'

Now -Ip • Up ~ _ 1/ilp tHUp

• vp tA 'tHbp _ I/Vp tA tHAvp'

Take H • A-I, this gives us

'4 • vptrp _ l/VptAvp•

If on the other ham! lie take sp • bp then

y - Axp • A~ • A(bp _ 1 - "pUp) • y - Axp _ 1 - tJrf'1>f and

A~ • ~ _ 1 + -IpA'1>' and

~ • ~ - 1 + -Ip'1>" Now -Ip. ~Hbp _ 1/'1> tA'1>'

9

ot the d.nation ot the approximation from the true solution.

Hence either rp or Sp can be taken as bp•

Take the matru A to be positive de.finitee AnY' system of

equations w • Bx, B nonsingu1ar, can be transformed by a premu1­

tip1ication of at on both sides into a system in which the coeffi­

cient matrix is positive definit",. The solution ot

y • at" • BtBx • Ax

as also the solution of w • Bx.

Let r p • ~, then

y - Axp • bp • bp _ 1 - ¢pUp • y - AXp _ 1 - 4UpJ and

A.xp • A~ _ 1 + tpup, and

~ • ~ _ 1 + sip!-l~.

Since up is an arbitrary set of vectors we may substi tute IIp. vp

to obtain

XI> • Xp - 1 + ripvp •

Now Ip • UptHl>p _ l/llptHUp

• vp t.A. tHbp _ l/VptA tHAVp.

Take H • A-I, this gives us

If on the other ham we take sp • bp then

y - A:l]:> • A"P • A(bp _ 1 - 'pUp) • y - Axp _ 1 - s1pAUp, and

A:xp • A:J.P _ 1 + rlpAUp, and

~ • XI> - 1 + rip~.avr rip· ~Hbp _ 1/'\ltA~.

10

Take H • A then Ip • IIptrp _ J.lu~up.

When the vector fpllp is subtracted !'rom %p the new residual rp

may be thought of as a leg of a right triangle of which rp _ 1 is the

hypotenuse {provided up and rp _ 1 were not orthogonal}.

Any rule for selecting the vectors IIp (or equivalently the vp)

at each step defines a particular iterative process; of these the

following three aEe in common use.

2. Ilethod of steepest descent. This method prescribes that we

take up • rp _ 1., Since A in the equation y - Ax • 0 will be taken to

be positive definite, this implies that the matrix A can be expressed

as the product of a matrix C multiplied by its transpose Ct , A • ctc,

where the matrix C is , nonsingular.

If we let Cx • z; then y • Ax • ctex • Ctz. Defining the function

g{x) we have,

g{x) - (Cx - z)t{ex - z)

• xtctex - xtctz - ztex + ztz

• xtctex - xtctz - xtctz + xtotCx

- 2{xt.lx - xty).

!his function being a sum of squares is ne'Ver negative and has a min­

iJmJm value of zero for J!: • A-ly • c-lz. The function g{x) is a func-. .

tion of m variables (I' fh, ••• ,'1m- By taking the partial deriva­

tives of g(x) with respect to these variables we find the coordinates

of the gradient vector. Then

g{x) - ~ ;11 { ~ aij ~"~l - 2~"liY + ztz , 1</ ,., )./ )'/ t"

and since y and z are Imown constants

10

Take H • A then I p • Uptrp _ lIu~up.

When the vector fpup is subtracted from Xp the new residual rp

may be thought of as a leg of a right triangle of which. rp _ 1 is the

hypotenuse (provided up am r p _ 1 were not orthogonal).

Any rule for selecting the vectors Up (or equivalently the vp )

at each step defines a particular iterative process; of these the

following three a!e in cOJIDD.on use.

2. Method of steeEest desce~~. This method prescribes that we

take up • r p _ 1.. Since A in the equation y - Ax • 0 will be taken to

be positi" definite, this implies that the matrix A can be expressed

as the product of a matrix e multiplied by its transpose Ct , A • etc,

where the matrix C is. nonsingular.

If we let ex • z; then y • Ax • ctex • Ctz. Defining the function

g(x) we have,

g(x) • (ex'!" z) t(cx - z)

• xtctcx - xtctz - ztcx + ztz

!his function being a sum of squares is never negative and has a ndn­

imum value of zero for " • A-ly • c-lz. The function g(x) is a func-

tion of m v: riables ¢lJ '12' ••• , '1m- By taking the part.ial deriva­

tives of g(x) with respect to these variables we find the coordinates

of the gradient vector. Then

g(x) ·~,'li~.~I aij ~~jl - 2~ ;liY + ztz ,

and s~.nce y and z are known constants

11

Now 111 • IjlJ therefore .... '" ~(x) • 2~~aij;E 'Ijl -y • 2(Ax - y).

" "" J ~ ' )_1

The function g(xl evaluated at the point x • ~ _ 1 is undergoing its

most rapid variation in the direction .rp _ 1 since,

gx(Xp - 1) • 2(Axp _ 1 - y) • 2(-rp _ 1) • -2rp - 1-

If we think of our problem as one of mLni.mizing. g(x} by suec.essin

steps, it is natural to take each step in the direction of most

rapid decrease_ Hence we have justified the selection of rp _ 1

as the vector ~ for most rapid minimLsation of the residuals.2

The method of steepest descent then proceeds as follOWl!l:

;y - AX() • rO

Xl • X() + tlrO where 1/1 • rOtrO/rotAro

y-AJ!:l·~

• • •

7'1' • Xp _ 1 + ¢prp _ 1 where rip • rpt_ lrp _ l/rpt_ lArp _ 1

;y - Axp • rp

• • •

Since this is a BeU correcting method it is desireable to S8C-

rifice temporar;y accurac;y for ultimate speed; an error of calculation

made in any step continually corrects itself in subsequent iterations.

n

mll'l m '" ",hl

~(x) .~ F' 8ij ~~jl + ~hi~, ;/81j - 2y.

Now '!U • ~jH therefore,." ... t>'l

fL. (x) • 2~~ 8ij ~ ~j1 -y • 2(Ax - y)."'X .~,J'/ ;,..

The funct.ion g(x) evaluated at the point x • ~ _ 1 is undergoing ita. .

DlOl!t rapid variation in the direct10nrp _ 1 einee,

gx(Xp - 1) • 2(Axp - 1 - y) • 2(-rp - 1) • -2rp - 1-

If we think of our problem as one of minimizing. g(x) by suec.essive

steps, it is natural to take each step in the direction of most

rapid decrease_ Hence we have justified the selection of r p ... 1

l!l the vector Up for most rapid minimization of the resid als.2

The method of steepest descent then proceeds 8S follows:

y - AX() • rO

Xl • XC + IIrO where t1 • rOtrO/rOtAro

•••

Top • xp _ 1 + ¢'prp _ 1 where 'p • r pt_ lrp _ Jlrpt_ lArp _ 1

y - AXp • r p

•••

Since this is a selt oorrecting method it is deBireable to S8C-

r1.flce t mporary accuracy for ultimate speed; an elTor of calculation

made in any step continually COIT9cts itself' in subsequent iterations.

12

3. Gauss-Seidel method. This method is the classical iterative

method for solving linear equations that feature symetry' and rela-

tively large diagonal terms, however it may also be used for certain

other systema of equations.

Usi.ng this method we let ~ • 1l.nn + i - et where v - 1, 2, • • • •

Then Xp differs f'r0l!l Xp _ 1 only in the i th element. In selecting up

we do so in such a manner that the ith element of rp (the ith residual)

:Us zero. This means that the ith equation is satisfied exactly. This

process 'frill require more steps than either of the other two methods;

however the sinpl1city of choice and the ease of computation give it a

great advantage in automatic machinery work.

In using this method for hand computation it is usually best to

set up a -At table and a residual table.3 The residuals are then suc­

cessibe1y reduced to zero in the following manner: divide the residual

ri to be eliminated by the corresponding main diagonal element of A,

aU, and use the quotient ipei (,fp - ei trp _ 1/0t tAot) as the change

in Xi. Change each of the residuals ~, r2, ••• , rm by ,fpBiaji

where j ·1, 2, ••• , m; that is, add ,fpei(-aji) to each of the

reSiduals. Each step of the calculation thereafter involves a s i ngle

application of an operation of the "basic kind". The residual table

merely records in tabular form the operation used, the _extent to which

it is used and the consequent effects on the residuals. One operation

of this type corresponds to each unknown in the problem and is called

3Al1en, Deryck N., Relaxation IIethods (New- York: IlcGraw-H111 Book Company, Inc., 1954), pp. 3-1.

12

3. Gauss-Seidel method. This method is the classical iterative

method ror ~olving linear equations that feature symetry and re18-

tively large diagonal terms, however it may also be used for certain

other systems of equations.

Usi,ng this method we let up • Uvm + i • e1 where v-I, 2, ••••

Then xp differs from Xp _ 1 only in the i tb element. In selecting up

we do so in such a JMnner that the ith element of rp (the ith residual)

is zero. This means that the i th equation is satie.fied exactly. Thie

process will require more steps than either of the other two methods;

however the sinplicity of choice and the ease of computation give ita

great advantage in automatic machinery work.

In using this method for hand computation it is usually best to

set up a -At table and a residual table.3 The residuals are then sue-

cessibely reduced to zero in the follOlrlng manner:: divide the residual

r i to be eliminated by the corresponding main diagonal element of A,

aii, and use the quotient ~pei (;p - ei trp _ l/~ tAei ) as the change

in Xi_ Change each of the residuals TI, r2, ••• , I'm by 'peiaji

where j • 1, 2, ••• , m; that is, add Ipei(-aji) to each of the

residuals. Each step of the calculation thereafter involves a single

application of an operation of the "basic kind". The residual table

merely records in tabular form the operation used, the extent to which

it is used and the consequent effects on the residuals. One operation

of this type corresponds to each unknown in the problem and is called

3Allen, Deryck N., Relaxation Methods (New Yorkr McGraw-Hill BookCompany, Inc., 1954), pp. 3-1.

13

a basic unit operation.

The residuals, one of which stan6s identically for the non-zero

side of each equation in the system are such that for any values

assigned to the original unknowns xl' ~, ••• , xm corresponding

values can be calculated for rl' r2' • • • , rm directly from the

defining relations. The method would then be self-cheeking. To

save time, however, use is made of the _At tabla and each of the resid-

uals ri is ~hanged by 'pei(-aji) as previously indicated. A cheek on

the residuals at periodic intervals and at the end of the computation

is therefore necessary.

Under certain circumstanees changing the coefficient of xt in

the ith equation to one is a useful preliminary step. The teehniques

for handling the computation in the Gauss-8eidel method also apply to

the handling of the eomputation in the following method.

4. Method of relaxation. This method is an adaptation of the

Gauss-5e1del method, and therefore always takes Up to be some 81. but

the selection is made as the computation progresses. Since the choiee

of up • Sf eliminates the ith element of rp one can ehoose to eliminate

the largest residual (or make this residual assume any desireable value

in view of future iterations). Selecting the largest residual is not

nece4sarily the best choice. The magnitude of the correction is

measured by the correction vector ¢pup and this magni tude is

Uptrp _ If Up tA Up}-l/2

Now when u • e this becomes Sf trp _ lfaii)-1/2. Hence one

should examine the. quotients of the residual components divided by

13

a basic unit operatt"on.

The residuals, one of which stan6s identically for the non-zero

sid of each equation in the system are such t~at for any values

assigned to the original unknowns xl' ~, ••• , ~ corresponding

values can be calculated for rl' r2' • • • , r m directly from the

defining relations. The method would then be self-ehecking. To

save time, however, use is made of the _At table and each of the resid-

uals ri is ~hanged by sipei (-aji) as previously indicated. A cheek on

the residuals at periodic intervals and at the end of the computation

is therefore necessary.

Under certain circumstances changing the coe.fflcient of Xi in

the i ttl equation to one is a useful prellmir'lary step. The techniques

for handling the computation in the Gau8s-8eidel method also apply to

the handling of the computati.on in the following method.

4. Method of relaxation. This method is an adapta t,c:.on of the

Gausa-Seidel method, and therefore alwa.Y.S takes Up ~o be some ei, but

the selection is made as the computation progresses. Since the choice

of up ~ eliminates the ith element of rp one can choose to eli~n8te

the largest residual (or ma~ this residual assume any desireable value

in view of future iterations). Selecting the largest residual is not

necessarily the best choice. The magnitude of the correction is

measured by the correction vector ~pup and this magnitude is

Upt rp _ l.fuptAUp)-1/2

Now when u • e this becomes ~ trp _ Ifaii)-1/2. Hence one

should examine the quotients of the residual components divided by

the corresponding -iiii and eliminate the largest quotient (in absolute

value).

Various refinements have been made on the method of relaxation

to make it more practical for hand computation. These refinements

include: overrelaxation, reduction of the residuals beyond the zero

value; underrelaxation; block relaxation, changing several variables

. by the same amount; and group relaxation, changing several variables

by ,varying amounts. Group relaxation is sometimes used to produce a

modification of the _At table that has a main diagonal of relatively

large elements. This table is const~Jcted with the idea that each

change in the unknowns li~ted will produce a relatively large change

in only one of the residuals and a different residual for each entry.

Relaxation with this table should show much quicker convergence to a

solution than with the -At table. • , .

5. Summary. USing the method of relaxation causes the xt's to

converge to a solution more rapidly than the method of Seidel. This

feature together with the advantages incurred with the use of under

and overrelaxation make this method the best of the three discussed

for systems with a "reasonable" number of unknowns. The ease of com­

putation distinguishes the Gauss-8eidel method. The method of steep-

est descent involves the most difficult computations of the three,

but relatively rapid convergence to a solution for systems in which

the number of unknowns is "large" is its reward. The method to use

in solving any particular system is then chosen with the special char-

acteristics of the system, the requirements of the solution and the

capabilities of the computor in mind.

the corresponding"l'i'ii and eliminate the largest quotient (in absolute

value).

Various refinements have been made on the method of relaxation

to make it more practical for hand computation. These refinements

include: overrelaxation, reduction of the residuals beyond the zero

value; underrelaxation; block relaxation, changing several variables

. by the same amount; and group relaxation, cha~ng several variables

by .varying amounts. Group relaxation is somet.imes used to produce a

modification of the _At table that has a main diagonal of relatively

large elements. This table is constl"Jct.ed with the idea that each

change in the unknowns listed will produce a relatively large change

in only one of the residuals and a different residual for each entry.

Relaxation with this table should show much quicker convergence to a

solution than with the -At table.. -

5. Summary. 'O'sing the method or relaxation causes the ~fS to

converge to a solution more rapidly than the method of Seidel. This

feature together "Wi th the advantages incurred with the use of under

and overrelaxation make this method the best of the three discussed

for systems nth a "reasonable" number of unlmowns. The ease of com-

putation distinguishes the Gaus8-Seid~1 method. The method of steep-

est descent irrvolves the most difficult computations of the three,

but relatively rapid convergence to a solution for 8ystems in which

the number of unknowns is ltlarge" is its reward. The method to use

iJ'l solving any particular system is then chosen with the special char-

acteristics of the system, the requirements of the eolution and the

capab11ities of the computor in mind.

15

In using an iterative method a computor ceases to look upon the

problem as that of finding those values of xl' X2. • • • , ~ which

satis~ the equations, but regards it instead as the determination of

a set of values which make the residuals zero.

Since each iteration is a check on the progress of the method,

it is usually possible to see when things are getting "out of hand".

After one or two iterations the changes in the computed values should

be gradual. Erratic changes will mean that the method is either not

applicable to this type of problem or that an error of some kind has

been made in obtaining the erratic value and therefore the value

whould be checked before proceeding further.

To illustrate the essential characteristics of each of the tp.ree

methods discussed the same problem will be worked by each method. The

last two methods described were originally self checking. but the adap­

tations made here were not, therefore a check was taken at periodic

intervals, that is, substitution of ~ was made in each of the original

equations and the values of the residuals so computed were taken as the

starting point of the next series of iterations.

Each method was begun with Xo • " for purposes of comparison. The

solution of this system (by direct methods) is ~1, 1, 2. The approximate

solutions achieved are such that they sre stable to three decimal places

(in the methods of Gauss-5eide1 and relaxation). This can be verified

by examination of the previous values for the unknowns and the observa­

tion that the total of the residuals whether added or subtracted from

the values os the unknowns would not influence the third deCimal place

15

In using an iterative method a computor ceases to look upon the

problem as that of finding those valup.s of xl' x2' • • • , Xm hleh

satisfy the equations, but regards it instead as the determination of

a set of values which make the residuals zero.

Since each iteration is a check on the progress of the method,

it is usually po!!sible to see when things are getting "out of hand".

After one or two iterations the changes in the computed values should

be gradual. Erratic changes w:i.l1 mean that the method is either not

applicable to this type of problem or that an error of some kind has

been made in obtaining the erratic value and therefore the value

whou1d be checked before proceeding further.

To i1lustra te the essential characteristics of each of the three

methods discussed the same problem will be worked by each method. The

last two methods described were originally self checking, but the adap­

tations made here were not, therefore a check was taken at periodic

intervals, that is, substitution of xp was made in each of the original

equations and the values of the residuals ao computed were taken as the

starting point of the next series of iterations.

Each method was begun With Xo • 0 for purposes of comparison. The

solution of this system (by direct methods) is ...1, 1, 2. The approximate

solutions achieved are such that they re stable to three decimal places

(in the m.ethods of Gau8s-8eidel and relaxa ti on). This can be verified

by examination of the previous values for the unknowns and the observa­

tion that the total of the residuals whether added or subtracted from

the values os the unknowns would not influence the third decimal place

16

of the unknowns.

Only seventeen steps were needed for this result ld. th the method

of relaxation while twenty steps were needed for the Gauss-Seidel

method. By under and ~rrelaxation the solution ll'I4y be arrived at much

faster; see the last exaft\)le. The method of steepest descent which CO\1-

verged ,so slowly in this example is really the fastest converging when

when n is "large".

In certain iterations the reSidual had to be changed by amounts

too large or too small. An autoll'l4tic II!Ichine would probably be designed

to change the residuals by too large an amount. A better proceedure

would be to consistently round the change in the variable to the nearest

even or odd figure. This occurred in the fourteenth iteration of the

Gauss-5eidel method. At times the values of the unknowns or the residuals

may even worsen rather than iuprove; see ' the sixteenth iteration of the

Gauss-5eidel method.

6. Exauples. Let the equations be:

fl(~,~~X3)-4+5;+ ~ -0

f 2(xl' "2, X3) -1 - Xl - Sx2 + 4x3 - 0

f 3(xl' X:!, x3 ) -13 - 2xl + 3X:! - 7x3 - 0

1-5 -1 01

A- lS-4 -2 -3 7 ~~ X - 2

• 3 (-1, 1, 2)

Since a positive definite matrix is needed for most applications of the

method of steepest descent we will premultiply both sides of the pre­

ceeding equation by At.

16

of the unknowns.

Only seventeen steps were needed for this result lri.th the method

of relaxation while twenty steps were needed tor the Gauss-Seidel

method. By under and O"¥errelaxation the solution !!'By be arrived 8t much

f'a8ter; see the last example. The method of s·~pest descent which con­

vetrged so slowly in this example 1s really the fastest donverging hen

when n is "large".

In certain iterationa the reSidual had to be changed by amounts

too large or too small. An automatic machine would probably be designed

to change the residuals by too large n amount. A better proceedure

would be to consistently round the ohange in the variable to the nearest

even or odd figure. This occurred in the fourteenth iteration of the

Gauss-Seidel lMthod. At times the values of the unknmms or the residuals

may even worsen rather than improve;. see'the sixteenth iteration of the

Gau8s-5eidel method.. .

6. Examples. Let the equations be:

£1(~, ~~ x3) • 4 + 5~ + ~ - 0

t'2(xl' x2' x3) -1 - Xl - ~2 + 4x3 • 0

f'3{xl , ~, x3) -13 - 2x1 + 3~ - 7x3 • 0

y - Ax • 0 y ·l-~13

-5 -1 0\A-I g-4

-2 -3 7 ~~x.. X23 (-1, 1, 2)

Since a po itive definite matrix is needed for most applications of the

method of steepest descent we will premultiply both sides of the pr ­

ceeding equation by At.

17

Lettillg II • At,- and B • At! we have an equation in lIhich the coeffi-

cient matrix B is positive defin1te:

z - Bx - o.

The solution of this equation is also the solution of the equation

y - Ax • o.

l~o 19 - -1~

B· 1974--5~ 1~ -5~ 65

The transformed equations being:

gl (;,.. ~. x~) • -47 -~Ox1 -1~ *.. 19X~ • 0

g2(x1' ~. x3) ~ -51 -19x1 -7~ + 5~x~ • 0

g~(xl' ~. ~) • 95 +1~1 +5~~ - 65x3 • o.

11

Letting z • Aty and B • AtA we have an equation in 'Which the eoeffi-

cient matrix B is positive definite:

z - Bx • o.

The solution of this equation is also the solution of the equation

y - Ax • o.

The transfonned equations being:

g1(;, x2 , x3) • -47 -3Ox1 -1~ ~1.lgX3 • 0

g2(xl~ x2' x3) • -51 -19x1 -7~ +53X3• 0

g3(xp ~, x3)· 95 "·1~1 +53~ - 65x3 • o.

II!!

The method of steepest descent

~ xl ~ x3 rp r1 r2 r3

XC 0 o· 0 ro -47 -51 95 stiro - .40369 - .43r04 .81596

xl - ;40369 - ;43804 .~1596 ~ -11.87926 32.33095 11.L!.!!lOO6 --2~ - .30079 1.03636 .36799 ~ - .7gL!M • 59S32 1.1S395 r2 -13.52258 -17.62121 35.63357

__ 3r2 - .11690 - ~15233 .30004 x3 - .90138 ;/j4599 1.49199 r3 -1.57659 12.19S43 5J:.332S

--4r3 - .05135 ~39730 .17696 x4 - .95273 ~84329 1.66g95 r4 4.39951 -6.S4724 14.063M

'5r4 ~OM93 - .07615 .15640 x5 - .90380 .76714 1.~535 r5 -1.60536 6.1L!739 .'74227

--(jr5 - ;03138 .12018 .01L!51 X6 - .9351g .88732 1.g3~6 r6 -2.M620 -1. 380M 5.60382

tl7r6 - .02790 - .01434 .05820 x7 ":' .96308 .!'!72~ 1.!'!9OO6 r7 - .52914 3.2951g .55g6()

tl8r7 - ;01042 ~06489 .01100 Xg - .97350 .93787 1.90906 r8 - .13659 1.79003 .3~39

--~g - ' .00270 .03540 .00721 ~ - .~923 .96571 1;',14851 r9 - .59S41 - .39614 1!72334

tl10r9 - .00625 - .00413 .01799 xIO - .99548 .96158 1.96650 1'].0 - .00!!l62 .9S170 .22260

1111'].0 - .00002 .00195 .00044 x11 - .99550 .96353 1.96694 rn - .03715 .86332 .29699

H~

The method of steepest descent

Xp xl x2 x3 rp rl r2 r3

Xo 0 0- 0 ro -41 -51 95tl1rO - .40369 - .43804 .81596

xl - .40,69 - ~43804 .~1596 I1. -11-.81926 32:.33095 11.40006t21"J. - .3eo79 1.03636 .36799

x2 - .n448 ~59832 1.~395 r2 -13.52258 -11.62121 35.63351rJ3r2 - .11690 - ~15233 .30804

x3 - .90138 ~L4599 1.49199 r3 ~1.57659 12.19e'43 5.43328rl4r3 - .05135 ~39730 .17696

x4 - .95273 ~g4329 1.6M95 r4 4.39951 -6.e'472l+ 14.06348'5r4 ~01.J~93 - .07615 .15640

x5 - .90380 .76714 1.~2535 r5 -1.60536 6.14739 .74227lye, - ~03138 ~12018 .01451

X6 - .93518 .88732 1.~3~6 r6 -2.68620 -1.3806S 5.60382rJ7r6 - .02790 - .01434 .05820

x7 ~ ~96308 .87298 1.89806 r7 - .,291.4 3.29518 .55860tlgr7 - .01042 ~064g9 .01100

Xg - .97350 .93787 1.90906 rg - .13659 1.79003 .3l439;C)I'8 -' .00270 .03540 .00721

x9 - .98923 .96571 1~94g51 r9 - .59841 - .39614 1.723-34rlI0r9 - .00625 - .00413 ~01799

x10 - .99548 •9615g 1.96650 rto - .00062 .98170 .22260111liO - .00002 .00195 .00044

XII - .99550 .96353 1.96694 iiI - .03715 .86332 .29699

19

The Gauss-8eidel method

lrt X:2 x~ ¢'p6,t r ~ r2 r~

0 0 0 ~ • x2 , · . x~ ·0 4 -1 1~ - .8 281 • -.8 0 - .2 1104

- .025 ~82 • - ~025 - ~025 0 11.~25 1.61~ ;t~· 1~617S6 - .025 6.1!71h4 - .00002

- .795 ~e1· ~005 o· 6;M6114 .00998 .78~~ . ~82 ~ ;gOg~ ~g08~ .~ 2.43488

1.96570 ~~ ~ ~~4784 .OO8~ 1.~914 0 - .95666 ~1 • - .16166 0 1.55~06 - .~2"2

.m4~ ~e2· .1941~ ~1941~ .00002 .25907 2.00271 9~ • . ' .0~701 .1941~ .14006 0

- .99549 r,0s:L • - ~O~gg~ -~00002 .18689 - .07766 1.00079 ~le2· ~02~~6 ;02~34 .00001 - .00758

2.oo16~ ~e~ • - .00108 ~02~34 - .004~1 - .00002 -1.00016 ~~e1 a - ;00467 - ;00001 .000~6 - .009~6

1.000/34 . ~e2· .00005 .00004 - .00004 - .00921

-1.00017' 2.ooo~1 ~5e, • ~ .001~2 .00004 - .005~2 ;OOOO~

~6el • - .00001 - .00001 - .005~1 .00001 1.00018 ' ~~ • - .00066 - .00067 - .OOOO~ - .00197

2.oooo~ 18e~ . - .00028 - .00067 - .00115 - .00001

-1.00004 ~981 " .0001~ - .00002 - .0012S ;00025 1.00002 2082 ~ - .00016 - .0001S 0 - .0002~

19

The Gauss~e1de1 method

~ ~ x3 ;p~ r r1 r2 r;

0 0 0 ~ • x2 • x3 • 0 4 -1 13- .g ~e1 • _. ~8 0 - .2 11.4

- .025 ~~ • - ~025 - ~025 0 11.3251.61786 ;r.e;. 1~617g6 - .025 6.~7144 - .00002

- .795 ~e1· ~005 O· 6~66WJ .OO99~.7g;; ~~ ~ ~80e; ~g08; .00004 2.4348S

1.96570 ~e;. ~34784 .80~3 1.3914 0- .95666 1J~1 • - ~16166 0 1.55306 - ~;23;2

.97743 ~e2· .1<;41; ~1941; .00002 .259072.002n 9~ • ..•03701 .1941; .14806 0

- .99549 ~0t!J. • - .038S3 - ~00002 .18689 - .077661.00079 ii1e2 • .02336 ~02334 .00001 - .00758

2.00163 ~e3 - .0010S ~02334 - .00431 - .00002-1.00016 ~3el z - ~00467 - ~OOOOI .000;6 - .00936

1.00084 - ~e2· .00005 .00004 - .00004 - .00921

-1.00017'2.000;1 ~5e; c ~ .00132 .00004 - .005;2 ~00003

~6el • - .00001 - .00001 - .00531 .000011.00018 ~-re2 - - .00066 - .00067 - .0000; - .00197

2.00003 IS 3 - ~OO02g - .00067 - .00115 - .00001

-1.00004 ~~ ~ .00013 - .00002 - .00128 ~OO0251.00002 20~ .. - .00016 - .00018 0 - .00023

2D

The method of relaxation

Xl ~ x~ ;pei r1 r2 ~

0 0 0 x .~·x -0 4 -1 1~ 1.S5714 ~e~. 1~e5714 4 6.42e56 .00002

.00~57 ~2- ~00~57 4.oo~57 o· 2.410n - .96071 ~e1 • - .96071 ~00002 .96071 .48931

.92~66 ~e2- .12009 .12011 - .00001 .S4958 1.97S51 :.ra~. .121~7 ~12011 .4e547 - .00001

.91"434 ~e2- .0606S .10079 ~OOOO~ .1e20~ 2.00451 t . .02600 .10079 .1040~ .00003

- .996S7 ~ei • - .0~616 - .00001 .]4019 - .07229 1.00le6 ge2- .01752 .01751 .oooo~ - .019n

2.00169 ~oe~ • - .002e2 .01751 - .01125 .00001 -1.ooo~7 . ~le1 • - ~00~50 .00001 - .00775 - .00699

1.000g9 ~e2 • - .00097 - .00096 .00001 -.00990 2.ooo2g ?,~e3 • - .00141 - .00096 - .0056~ - .oooo~

1.00019 - ~e2 • - .00070 - .00166 - .OOOO~ - .0021~ 1.~ ?,5e3 • - .OOO~O - .00166 - .OO12~ - .oooo~

-1.00004 ?,6e1. .OOO~~ - ~OOOOl - .00156 .0006~ .99999 11'2 • - .00020 - .00021 .00004 .OOOO~

The method of re~tion with under and over-relaxation

Xl ~ x~ ;pei r1 r2 r~

0 0 0 (i.~.x~.o 4 -1 1~ 2 ~e~. 2 4 7 -1

1 ~. 1 5 -1 2 -1 ~~ ·-1 0 0 0

2D

The mthod of relaxation

xl ~ x:; ;pei r'1 r2 r"3

0 0 0 x .~·x -0 4 -1 131.85714 ~e:;. l~g~714 4" 6.42856 .00002

.S0357 ~2" ~go357 4.00357 0" 2.41073- .960n ·3el .. - .96071 ~OOOO2 .96071 ~931

.92366 ~e2" .12009 .12011 - .00001 .84.95g1.9~1 ~3" .12137 ~12011 .48547 - oo1סס.

.98434 ~e2· .06068 ~18079 ~OOOO3 .1820:;2.00451 of3· .02600 .18079 .1040:; .00003

- .~7 ~el III - ~03616 - .00001 .14019 - .072291.0011l6 ge2 - .01752 .01151 .00003 - .01973

2.00169 ~oe3 • ~ .00282 .01751 - .01125 .00001-1.00037 . ~lel .. - ~O0350 oo1סס. - .00775 - .00699

1.00089 ~e2 - - .00097 -.00096 .00001 - .009902.0002g ¢J,'3e3 • - .00l41 - .00096 - .00563 - ~OOOO3

1.00019· .. " ~e2 .. - .00070 - .00166 - ~OOOO; - .002131.99998 ~5e; .. - .00030 - ~00166 - .00123 - .00003

-1.00004 ~6e1" ~OOO33 - .00001 - .00156 .00063.99999 1,.,e2 • - .00020 - .00021 .00004 .00003

The method of relaxation with under and over-re1aocartions

Xl x2 %3 tip s i 1'1 r2 1':;

0 0 0 Ii .. ~ .. %3 .. 0 4 -1 132 ~e3 .. 2 4 7 -1

1 ;;~ .. 1 5 -1 2-1 3€I • -1 0 0 0

BIBLlOORAPHY

~llen, Deryck N~ Relaxation Methods. New York: McGraw Hill Book Company, Inc., 1954.

Buckingham, Richard A~ Numerical Methods. London: Sir Isaac l'i tman and Sons, Ltd., 1957.

Dwyer, .Paul S. Linear Computations. New York: John Wiley and Sons, Inc., 1951.

Hote1ling, Harold. Anal sis of 8 co lex of statistical variables into frinCiP81 components, Journa of Educationa Psychology, Vol.

OCtober, 1933) pp. 502-4.

Rotelling, Hsrold. Some new methods in matrix calculation, Annals of Mathematical Statistics, Vol. 14 (1943) pp. 1-34.

Householder, Alston S. Pr; ncipals of Numerical Analysis. New York: McGraw-Hill Book Company, Inc., 1953. .

PerUs, Sam. The0rr, of ' Matrices. Cambridge, Massachusetts: Addison-Wesley Press, Inc., 952.

Scarborough, James B. Numerical Mathematical Analysis. Baltimore, Ohio: Johns Hopkins Press, 1955.

Wittaker, Edmund T., and RObinson, George. The Calculus of Observations. London: Blackie and Son, Lim! ted, 1926.

21

\

\

BIBLlOORAPHY

Allen, Deryck N~ Relaxation Methods. New York: McGraw Hill Book Company,Inc., 1954.

Buckingham, Richard A.~ Numerical Methods. London: Sir Isaac Pitman andons, Ltd., 1957.

Dwyer, Paul S. Linear Computations, New York: John Wiley and SOM, Inc.,1951.

Rotelling, Harold. Analysis of a complex of statistical variables intotrinCipa1 components, Journal of Educational Psychology, Vol. 24

OCtober, 1933) pp. 502-4.

Rotelling, Harold. Some new methods in matrix calculation, Annals ofMathematical Stati~tics, Vol. 14 (l943) pp. 1-34.

Householder, Alston S. Principals of Numerical Analysis. New York: McGralf-. Hi11 Book Company, Inc., 1953. . "PerIis, Sam. TheorY of "Matrices. Cambridge, Massachusetts: Addison-Wesley

Press, Inc.,~952.

Scarborough, James B. Numerical Mathematical Analysis. Baltimore, Ohio:Johns Hopkins Press, 1955.

Wittaker, Edmund T., and Robinson, George. The calculus of Observations.Londom Blackia and Son, Limited, 1926.

21