Upload
maths203
View
222
Download
0
Embed Size (px)
Citation preview
7/29/2019 Numerical Analysis 323
1/161
Computational Linear Algebra Syllabus
NUMERICAL ANALYSIS
Linear Systems of Equations and Matrix Computations
Module1: Direct methods for solving linear system of equationSimple Gaussian elimination method, gauss elimination method with partial pivoting,determinant evaluation, gauss Jordan method, L U decompositions Doolittles lu
decomposition, Doolittles method with row interchange.
Module 2: Iterative methods for solving linear systems of equations
Iterative methods for the solution of systems equation, Jacobin iteration,gauss seidel
method, successive over relaxation method (sort method).
Module 3: Eigenvalues and Eigenvectors
An introduction, eigenvalues and eigenvectors, similar matrices,hermitian matrices,
gramm Schmidt orthonormalization,vector and matrix norms.Module 4: Computations of eigenvaues
Computation of eigenvalues of a real symmetric matrix, determination of the eigenvaluesof a real symmetric tridiagonal matrix, tridiagonalization of a real symmetric matrix,
Jacobin iteration for finding eigenvalues of a real symmetric matrix, the q r
decomposition, the Q-R algorithm.
Vittal Rao/IISc, Bangalore V1/1-4-04/1
7/29/2019 Numerical Analysis 323
2/161
Computational Linear Algebra Syllabus
Lecture Plan
Modules Learning Units Hours perTopics
TotalHours
1. Simple Gaussian elimination method 1
2. Gauss elimination method with partial
pivoting.
2
3. Determinant evaluation 1
4. Gauss Jordan method 1
5. L U decompositions 1
6. Doolittles LU Decomposition 2
1. Direct
methods for
solving linearsystem of
equation.
7. Doolittles method with row interchange. 2
10
8. Iterative methods for the solution ofsystems equation
3
9.Jacobi iteration. 210.Gauss Seidel method 2
2. Iterativemethods for
solving linearsystems of
equations.
11.Successive over relaxation method (sortmethod).
2
9
12.An introduction. 113.Eigenvalues and eigenvectors, 214.Similar matrices, 215.Hermitian matrices. 116.Gramm Schmidt orthonormalization, 2
3. Eigenvaluesand Eigenvectors
17.Vector and matrix norms. 1
9
18.Computation of eigenvalues 219.Computation of eigenvalues of a real
symmetric matrix.2
20.Determination of the eigenvalues of a realsymmetric tridiagonal matrix,
2
21.Tridiagonalization of a real symmetricmatrix
1
22.Jacobian iteration for finding eigenvaluesof a real symmetric matrix
1
4. Computations
of eigenvalues.
23.The Q R decomposition 1
11
Vittal Rao/IISc, Bangalore V1/1-4-04/2
7/29/2019 Numerical Analysis 323
3/161
Computational Linear Algebra Syllabus
24.The Q-R algorithm. 2
Vittal Rao/IISc, Bangalore V1/1-4-04/3
7/29/2019 Numerical Analysis 323
4/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
1.DIRECT METHODS FOR SOLVINGLINEAR SYSTEMS OF EQUATIONS
1.1.SIMPLE GAUSSIAN ELIMINATION METHODConsider a system of n equations in n unknowns,
a11x1 + a12x2 + . + a1nxn = y1
a21x1 + a22x2 + . + a2nxn = y2
an1x1 + an2x2 + . + annxn = yn
We shall assume that this system has a unique solution and proceed to describe thesimple Gaussian elimination method for finding the solution. The method reduces thesystem to an upper triangular system using elementary row operations (ERO).
Let A(1) denote the coefficient matrix A.
A(1) = Where a
nnnn
n
n
aaa
aaa
aaa
)1(2
)1(1
)1(
2)1(
22)1(
21)1(
1)1(
12)1(
11)1(
......
.......................
......................
.....
.....
(1)ij = aij
Let
y(1) = Where y
(1 )
1
(1 )
2
(1 )
n
y
y
y
M (1)
i = yi
We assume a(1)11 0
Then by ERO of type applied to A(1)
reduce all entries below a(1)
11 to zero. Let the
resulting matrix be denoted by A
(2)
.
A(1)
+ 11
)1(RmR ii
A(2)
Where ;11
)1(
1)1(
1)1(
a
am
ii = i > 1.
Note A(2)
is of the form
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/1
7/29/2019 Numerical Analysis 323
5/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
M1/L1and L2/V1/May2004/2
A(2) =
nnn
n
n
n
aa
aa
aa
aaa
)2(2
)2(
3
)2(
32
)2(
2)2(
22)2(
1)1(
12)1(
11)1(
......0
.....................0
......0
......
Notice that the above row operations on A(1) can be effected by premultiplying A(1) by
M(1)
where
M(1) =
)1(
1
1
)1(
31
)1(
21
00001
n
n
m
Im
m
M
i.e.
M(1)
A(1)
= A(2)
Let
y(2) = M(1) y(1) i.e)2()1( 11 yy
RmR ii +
Then the system Ax = y is equivalent to
A(2)x = y(2)
Next we assume
a(2)22 0
and reduce all entries below this to zero by ERO
A(2) + 2
)2(ii mR
A(3) ; ;22
)2(
2)2(
2)2(
a
am
ii = i > 2.
Here
1 0 0 . . . 0
0 1 0 . . . 0
0 m(2)32
VittalRao/IISc, Bangalore
M(2) = 0 m(2)42 In-2
7/29/2019 Numerical Analysis 323
6/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
.
0 m(2)n2
and M(2) A(2) = A(3) ; M(2) y(2) = y(3) ;
and A(3) is of the form
=
nnn
n
n
n
aa
aa
aaa
aaa
A
)3(3
)3(
3)3(
33)3(
2)2(
23)2(
22)2(
1)1(
12)1(
11)1(
)3(
...00
...
...00
...0
......
MMMM
We next assume a
(3)
33
0 and proceed to make entries below this as zero. We thus getM(1)
, M(2)
, . , M(r)
where
1 0 . 0
0 1 . 0
0 1
m(r)
r+1r
M(r) = rxr m(r)r+2r In-r
.
.
.
m(r)nr
==
++
+
++
+++
+
nnr
nrr
nrr
rrr
rnr
rrr
n
n
rrr
aa
aa
aa
aa
aa
AAM
)1(1
)1(
1)1(
11)1(
)()(
2)2(
22)2(
1)1(
11)1(
)1()()(
...000
.........
...0
......0
.........0
............
MMM
MM
M
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/3
7/29/2019 Numerical Analysis 323
7/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
M(r) y(r) = y(r+1)
At each stage we assume a(r)rr 0.
Proceeding thus we get,
M(1), M(2), . , M(n-1) such that
M(n-1) M(n-2) . M(1) A(1) = A(n) ; M(n-1) M(n-2) . M(1) y(1) = y(n)
a(1)11 a(1)
12 . . . . . a(1)
1n
a(2)22 . . . . . a(2)
2n
where A(n) = .
.
a(n)nn
which is an upper triangular matrix and the given system is equivalent to
A(n)x = y(n)
and since this is an upper triangular, this can be solved by backward substitution; andhence the system can be solved easily
Note further that each M(r) is a lower triangular matrix with all diagonal entries as 1.
Thus let M(r) is 1 for every r. Now,
A(n) = M(n-1) . M(1) A(1)
Thus
det A(n)
= det M(n-1)
det M(n-2)
. det M(1)
det A(1)
det A(n) = det A(1) = det A since A = A(1)
Now A(n) is an upper triangular matrix and hence its determinant is
a(1)11 a(2)
22. a(n)nn. Thus det A is given by
det A = a(1)11 a(2)
22. a(n)nn
Thus the simple GEM can be used to solve the system Ax = y and also to evaluate det A
provided a(i)ii 0 for each i.
Further note that M(1), M(2), . , M(n-1) are lower triangular, and nonsingular as their det =
1 0. They are all therefore invertible and their inverses are all lower triangular, i.e. if L= M(n-1) M(n-2) . M(1) then L is lower triangular, and nonsingular and L-1 is also lower
triangular.
Now LA = LA(1) = M(n-1) M(n-2) . M(1) A(1) = A(n)
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/4
7/29/2019 Numerical Analysis 323
8/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
Therefore A = L(-1) A(n)
Now L(-1) is lower triangular which we denote by and A(n) is upper triangular which wedenote by u, and we thus get the so called u decomposition
A = u
of a given matrix A as a product of a lower triangular matrix with an upper triangularmatrix. This is another application of the simple GEM. REMEMBER IF AT ANY
STAGE WE GET a(1)ii = 0 WE CANNOT PROCEED FURTHER WITH THE SIMPLE
GSM.
EXAMPLE:
Consider the system
x1 + x2 + 2x3 = 4
2x1 - x2 + x3 = 2
x1 + 2x2 = 3
Here
=
021
112
211
A
=
3
2
4
y
)2(2
)1(
210
330
211
021
112
21112
13
AARR
RR
=
=
a(1)11 = 1 0 m(1)
21 = -2 a(2)
22 = -3 0
m(1)31 = -1
=
101
012
001)1(
M )1()1()2()1(
1
6
4
3
2
4
yMyy ==
=
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/5
7/29/2019 Numerical Analysis 323
9/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
)3(3
1
)2(
210
330
21123
AA
RR
=
+
a(3)
33 = -3
M(2)31 = 1/3
=
13
10
010
001)2(
M
==
3
6
4)2()2()3(
yMy
Therefore the given system is equivalent to A(3)x = y(3)
x1 + x2 + 2x3 = 4
-3x2 - 3x3 = -6
- 3x3 = -3
Backward Substitution
x3 = 1
-3x2 - 3 = - 6 -3x2 = -3 x2 = 1
x1 + 1 + 2 = 4 x1 = 1
Thus the solution of the given system is,
=
=
1
1
1
3
2
1
x
x
x
x
The determinant of the given matrix A is
a(1)11 a(2)
22 a(3)
33 = (1) (-3) (-3) = 9.
Now
=
101
012
001)1(
1M
=
13
10
010
001)1(
2M
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/6
7/29/2019 Numerical Analysis 323
10/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
L = M(2) M(-1)
L-1 = (M(2) M(1))-1 = (M(1))-1 (M(2))-1
=
101
012001
13
10
010001
L = L(-1) =
13
11
012
001
u = A(n) = A(3) =
300
330
211
Therefore A = lu i.e.,
021
112
211
=
13
11
012
001
300
330
211
is the lu decomposition of the given matrix.
We observed that in order to apply simple GEM we need a(r)rr 0 for each stage r. Thismay not be satisfied always. So we have to modify the simple GEM in order to
overcome this situation. Further, even if the condition a(r)rr 0 is satisfied at each stage,simple GEM may not be a very accurate method to use. What do we mean by this?
Consider, as an example, the following system:
(0.000003) x1 + (0.213472) x2 + (0.332147) x3 = 0.235262
(0.215512) x1 + (0.375623) x2 + (0.476625) x3 = 0.127653
(0.173257) x1 + (0.663257) x2 + (0.625675) x3 = 0.285321
Let us do the computations to 6 significant digits.
Here,
A(1) =
625675.0663257.0173257.0
476625.0375623.0215512.0
332147.0213472.0000003.0
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/7
7/29/2019 Numerical Analysis 323
11/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
y(1) = a
285321.0
127653.0
235262.0(1)
11 = 0.000003 0
3.71837000003.0
215512.0
11)1(
21)1(
21)1( ===
a
aM
3.57752000003.0
173257.0
11)1(
31)1(
31)1( ===
a
aM
M(1) = ; y
103.57752
013.71837
001(2) = M(1) y(1) =
6.13586
5.16900
235262.0
A(2) = M(1) A(1) =
7.191818.123270
0.238609.153340
332147.0213472.0000003.0
a(2)22 = - 15334.9 0
803905.09.15334
8.12327
22)2(
32)2(
32)2( =
==
a
aM
M(2) =
1803905.00
010
001
y(3) = M(2) y(2) =
20000.0
5.16900
235262.0
A(3) = M(2) A(2) =
50000.000
0.238609.153340
332147.0213472.0000003.0
Thus the given system is equivalent to the upper triangular system
A(3)x = y(3)
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/8
7/29/2019 Numerical Analysis 323
12/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
Back substitution yields,
x1 = 0.40 00 00
x2 = 0.47 97 23
x3 = -1.33 33 3
This compares poorly with the correct answers (to 10 digits) given by
x1 = 0.67 41 21 46 9
x2 = 0.05 32 03 93 39.1
x3 = -0.99 12 89 42 52
Thus we see that the simple Gaussian Elimination method needs modification in order to
handle the situations that may lead to a(r)rr = 0 for some r or situations as arising in the
above example. In order to do this we introduce the idea of Partial Pivoting. The idea ofpartial pivoting is the following:
At the r th stage we shall be trying to reduce all the entries below the r th diagonal as zero.
Before we do this we look at the entries in the r th diagonal and below it and then pick
the one that has the largest absolute value and we bring it to the rth
diagonal position bya row interchange, and then reduce the entries below the r th diagonal as zero. When we
incorporate this idea at each stage of the Gaussian elimination process we get the GAUSSELIMINATION METHOD WITH PARTIAL PIVOTING. We now illustrate this with afew examples:
Example:
x1 + x2 + 2 x3 = 4
2x1 x2 + x3 = 2
x1 + 2x2 = 3
We have
Aavg =
3
2
4
021
112
211
1st Stage: The pivot has to be chosen as 2 as this is the largest absolute valued entry in the
first column. Therefore we do
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/9
7/29/2019 Numerical Analysis 323
13/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
Aavg 12R
3
4
2
021
211
112
Therefore we have
M(1) = and M
100
001
010
(1) A(1) = A(2) =
021
211
112
M(1) A(1) = y(2) =
3
4
2
Next we have
R2 R1 2 -1 1 2
A(2)avg 0 3/2 3/2 3
R3 R1 0 5/2 -1/2 3
Here
M(2) =
102
1
012
1001
; M(2)
A(2)
= A(3)
=
21
250
23
230
112
M2 y(2) = y(3) =
2
3
2
Now at the next stage the pivot is 2
5
since this is the entry with the largest absolute valuein the 1st column of the next sub matrix. So we have to do another row interchange.
Therefore
2 -1 1 2
A(3)avg 0 5/2 -1/2 2 23R
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/10
7/29/2019 Numerical Analysis 323
14/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
0 3/2 3/2 3
M(3) = M
010
100
001(3) A(3) = A(4) =
23
230
2
1
2
50
112
M(3) y(3) = y(4) =
3
2
2
Next we have
2 -1 1 2
A(4)
avg 23
5
3RR
0 5/2 -1/2 2
0 0 9/5 9/5
Here
M(4)
=
15
30
010
001
M(4) A(4) = A(5) =
5
900
2
1
2
50
112
M(4) y(4) = y(5) =
59
2
2
This completes the reduction and we have that the given system is equivalent to the
system
A(5)x = y(5)
i.e.
2x1 x2 + x3 = 2
2
5x2 -
2
1x3 = 2
5
9x3 =
5
9
We now get the solution by back substitution:
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/11
7/29/2019 Numerical Analysis 323
15/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
The 3rd equation gives,
x3 = 1
using this in second equation we get
2
5x2 -
2
1= 2
giving2
5x2 =
2
5
and hence x2 = 1.
Using the values of x1 and x2 in the first equation we get
2x1 1 + 1 = 2 giving x1 = 1
Thus we get the solution of the system as x1 = 1, x2 = 1, x3 = 1; the same as we had
obtained with the simple Gaussian elimination method earlier.
Example 2:
Let us now apply the Gaussian elimination method with partial pivoting to the following
example:
(0.000003)x1 + (0.213472)x2 + (0.332147) x3 = 0.235262
(0.215512)x1 + (0.375623)x2 + (0.476625) x3 = 0.127653
(0.173257)x1 + (0.663257)x2 + (0.625675) x3 = 0.285321,
the system to which we had earlier applied the simple GEM and had obtained solutions
which were for away from the correct solutions.
Note that
A =
625675.0663257.0173257.0
476625.0375623.0215512.0
332147.0213472.0000003.0
y =
285321.0
127653.0
235262.0
We observe that at the first stage we must choose 0.215512 as the pivot. So we have
A(1) = A A 12R (2) =
625675.0663257.0173257.0
332147.0213472.0000003.0
476625.0375623.0215512.0
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/12
7/29/2019 Numerical Analysis 323
16/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
y(1) = y y 12R (2) = M
285321.0
235262.0
127653.0(1) =
100
001
010
Next stage we make all entries below 1st diagonal as zero
R2 + m21R1 0.215512 0.375623 0.476625
A(2) A(3) 0 0.213467 0.332140
R3 + m21R1 0 0.361282 0.242501
Where
m21 = -11
21
a
a= -
215512.0
000003.0= - 0.000014
m31 = -11
31
a
a= -
215512.0
173257.0= - 0.803932
M(2) = ; y
10803932.0
01000014.0
001
(2) = M(2) y(2) =
182697.0
235260.0
127653.0
In the next stage we observe that we must choose 0.361282 as the pivot. Thus we have to
interchange 2nd and 3rd row. We get,
A(3) A 23R (4) =
332140.0213467.00
242501.0361282.00
476625.0375623.0215512.0
M(3) = y
010
100
001
(4) = M(3) y(3) =
235260.0
182697.0
127653.0
Now reduce the entry below 2nd diagonal as zero
A(4) A + 2323 RMR 5 =
188856.000
242501.0361282.00
476625.0375623.0215512.0
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/13
7/29/2019 Numerical Analysis 323
17/161
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
M32 = -361282.0
213467.0= - 0.590860
M(4) = y
159086.0
010
001
(5) = M(4) y(4) =
127312.0
182697.0
127653.0
Thus the given system is equivalent to
A(5) x = y(5)
which is an upper triangular system and can be solved by back substitution to get
x3 = 0.674122
x2 = 0.053205 ,
x1 = 0.991291which compares well with the 10 decimal accurate solution given at the end of page 9.
Notice that while we got very bad errors in the solutions while using simple GEMwhereas we have come around this difficulty by using partial pivoting.
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/14
7/29/2019 Numerical Analysis 323
18/161
Numerical analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L3/V1/May 2004/1
DETERMINANT EVALUATION
Notice that even in the partial pivoting method we get Matrices
M(k), M(k-1) . M(1) such that
M(k), M(k-1) . M(1) A is upper triangular and therefore
det M(k), det M(k-1) . det M(1) det A = Product of the diagonal entries in the final upper
triangular matrix.
Now det M(i) = 1 if it refers to the process of nullifying entries below a diagonal to zero;
and
det M(i) = 1 if it refers to a row interchange necessary for a partial pivoting.
Therefore det M(k) . det M(1) = (-1)m where m is the number of row inverses effected in
the reduction.
Therefore det A = (-1)
m
product of the diagonals in the final upper triangular matrix.In our example 1 above, we had M(1), M(2), M(3), M(4) of which M(1) and M(3) referred to
row interchanges. Thus therefore there were to row interchanges and hence
det A = (-1)2 (2)(2
5)(
5
9) = 9.
In example 2 also we had M(1), M(3) as row interchange matrices and
therefore det A = (-1)2 (0.215512) (0.361282) (0.188856) = 0.013608
LU decomposition:
Notice that the M matrices corresponding to row interchanges are no longer lower
triangular. (See M(1) & M(3) in the two examples.) Thus,
M(k) M(k-1) . . . . . M(1)
is not a lower triangular matrix in general and hence using partial pivoting we cannot get
LU decomposition in general.
7/29/2019 Numerical Analysis 323
19/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L3/V1/May 2004/1
GAUSS JORDAN METHOD
This is just the method of reducing Aavg to (AR / yR ) where AR = In is the Row Reduced
Echelon Form of A (in the case A is nonsingular). We could also do the reduction here by
partial pivoting.
Remark:
In case in the reduction process at some stage if we get arr = ar+1r = . . . . = ar+1n = 0, then
even partial pivoting does not being any nonzero entry to rth
diagonal because there is no
nonzero entry available. In such a case A is singular matrix and we proceed to the RRE
form to get the general solution of the system. As observed earlier, in the case A is
singular, Gauss-Jordan Method leads to AR = In and the product of corresponding M(i)
give us A-1
.
7/29/2019 Numerical Analysis 323
20/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/1
LU decompositions
We shall now consider the LU decomposition of matrices. Suppose A is an nxn matrix.
If L and U are lower and upper triangular nxn matrices respectively such that A = LU.
We say that this is a LU decomposition of A. Note that LU decomposition is not unique.
For example if A = LU is a decomposition then A = L U is also a LU decompositionwhere 0 is any scalar and L = L and U = 1/ U.
Suppose we have a LU decomposition A = LU. Then, the system, Ax = y, can be solved
as follows:
Set Ux = z (1)
Then the system Ax = y can be written as,
LUx = y,
i.e.,
Lz = y ..(2)Now (2) is a triangular system infact lower triangular and hence we can solve it by
forward substitution to get z.
Substituting this z in (1) we get an upper triangular system for x and this can be solved by
back substitution.
Further if A = LU is a LU decomposition then det. A can be calculated as det. A = det. L .det. U = l11 l22 .lnn u11u22 ..unn
Where lii are the diagonal entries of L and u ii are the diagonal entries of U.
Also A-1 can be obtained from an LU decomposition as A-1 = U-1 L-1.
Thus an LU decomposition helps to break a system into Triangular system; to find the
determinant; and to find the inverse of a matrix.
We shall now give methods to find LU decomposition of a matrix. Basically, we shall beconsidering three cases. First, we shall consider the decomposition Tridiagonal matrix;
secondly the Doolittless method for a general matrix, and thirdly the Choleskys method
for a symmetric matrix.
7/29/2019 Numerical Analysis 323
21/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/2
1 TRIDIAGONAL MATRIX
Let
=
bc
abc
abc
abc
ab
A
i
iii
1
12
432
321
21
0........0
0....0
........................
........................
0....0
0....0
0....00
be an nxn tridiagonal matrix. We seek a LU decomposition for this. First we shall givesome preliminaries.
Let i denote the determinant of the ith
principal minor of A
ii
ii
i
bc
abc
abc
ab
1
112
321
21
0....0
....0
....................
....................
0....
0....0
=
Expanding by the last row we get,
i = bii-1 ci-1 aii-2 ; I = 2,3,4, ..
..(I)
i = b1
We define i = 1
From (I) assuming that i are all nonzero we get
1
21
1
=i
i
iii
i
iacb
setting ii
ik=
1
this can be written as
).(..............................1
1 IIk
ackb
i
iiii
+=
7/29/2019 Numerical Analysis 323
22/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/3
Now we seek a decomposition of the form A = LU where,
=
1........00
........................
0....010
0........01
0............01
1
2
1
nw
w
w
L ;
=
n
nn
u
u
u
u
U
0....00
....00
....................
0....0
0....0
1
32
21
i.e. we need the lower triangular and upper triangular parts also to be tridiagonal
triangular.
Note that if A = (Aij) then because A is tridiagonal, Aij is nonzero only when i and j differby 1. i.e. only Ai-1i, Aii, Aii+1 are nonzero. In fact,
Ai-1i = ai
Aii = bi .. (III)
Ai+1i = ci
In the case of L and U we have
Li + 1i = wi
Lii = 1 .. (IV)
Lij = 0 if j>i or jj or i
7/29/2019 Numerical Analysis 323
23/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/4
Therefore
iiiiiiULa == 111
Therefore
i = ai .. (VII)
This straight away gives us the off diagonal entries of U. From (VI) we also get
=
=n
k
kiikii ULA1
iiiiiiiiULUL += 11
Therefore
)......(..........1 VIIIuwb iiii +=
From (VI) we get further,
=
++ =n
k
kikiii ULA1
11
iiiiiiii ULUL 1111 ++++ +=
iii uWc =
Thus (IX)iii uWc =
Using (IX) in (VIII) we get (also using I = ai)
i
i
ii
i uu
acb +=
1
1
Therefore
)..(....................1
1 Xu
acub
i
ii
ii
+=
Comparing (X) with (II) we get
).(....................1
XIkui
iii
==
7/29/2019 Numerical Analysis 323
24/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/5
using this in (IX) we get
).......(....................1 XIIc
u
cw
i
ii
i
i
i
==
From (VII) we get
I = ai .(XIII)
(XI), (XII) and (XIII) completely determine the matrices L and U and hence we get the
LU decomposition.
Note : We can apply this method only when I are all nonzero. i.e. all the principalminors have nonzero determinant.
Example:
Let
=
13000
13900
02520
00112
00022
A
Let us now find the LU decomposition as above.
We have
b1 = 2 b2 = 1 b3 = 5 b4 = -3 b5 = -1
c1 = -2 c2 = -2 c3 = 9 c4 = 3a2 = -2 a3 = 1 a4 = -2 a5 = 1
We have
0 = 1
1 = 2
2 = b2 1 a2 c10 = 2-4 = -2
3 = b32 a3 c21 = (-10) (-2) (2) = -6
4 = b43 a4 c3 2= (-3) (-6) (-18) (-2) = -18
5 = b54 a5 c43 = (-1) (-18) (3) (-6)= 36.
Note 1,2,3,4,5 are all nonzero. So we can apply the above method.
Therefore by (XI) we get
7/29/2019 Numerical Analysis 323
25/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
M1/L5/V1/May 2004/6
32
6;1
2
2;2
2
33
1
22
0
11 =
===
====
uuu
36
183
44 =
==
u ; and 2
1836
4
55 ===
u
From (XII) we get From (XIII) we get
12
2
1
11 =
==
u
cw 222 == a
2
1
2
2
22 =
==
u
cw 133 == a
33
9
3
33 ===
u
cw 244 == a
13
3
4
44 ===
u
cw 155 == a
Thus;
=
11000
01300
00120
00011
00001
L ;
=
20000
13000
02300
00110
00022
U
In the above method we had made all the diagonal entries of L as 1. This will facilitate
solving the triangular system LZ = y (equation (2)) in page 17. However by choosing
these diagonals as 1 it may be that the u i, the diagonal entries in U are small thus creatingproblems in backward substitution for the system Ux = z (equation (1) on page 17). In
order to avoid this situation Wilkinson suggests that in any triangular decomposition
choose the diagonal entries of L and U to be of the same magnitude. This can beachieved as follows:
We seek
A = LU
where
VittalRao/IISc, Bangalore
7/29/2019 Numerical Analysis 323
26/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/7
l1
L = w1 l2
;
wn-1ln
=
n
nn
u
u
u
u
U
0....00
....00
....................
........0
0....0
1
32
21
Lii = li
Now Li+1i = wi
Lij = 0 if j>i or jj; or j>i and j-i 2
Now (VII), (VIII) and (IX) change as follows:
iii Aa 1=
ki
n
kki UL
= 11 iiii UL 111
= iil
1
=
Therefore
ai = li-1I. (VII`)
iii Ab = kin
k
ikUL
=1
iiiiiiiiULUL += 11 iiii ulW += 1
`).......(..........1 VIIIulWb iiiii +=
iii Ac 1+= kin
k
ki UL += 1 1 iiii UL 1+= iiuw=
iii uwc = .. (IX`)From (VIII`) we get using (VII`) and (IX`)
7/29/2019 Numerical Analysis 323
27/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/8
ii
i
i
i
i
i ull
a
u
cb +=
11
1 .
ii
ii
ii ululca +=
11
1
`)..(....................1
1 Xpp
cab i
i
ii
i +=
where
pi = li ui
Comparing (X`) with (II) we get
1
==i
iii kp
therefore
1
=i
iiiul
we choose1
=i
iil
. (XIV)
=
1
sgni
iiu
1i
i
(XV)
Thus li and ui have same magnitude. These then can be need to get wi and i from (VII`)and (IX`). We get finally,
1
=i
iil
;
11
.sgn
=
i
i
i
i
iu
. . . . . . . . . . . .(XI`)
i
ii u
Cw = . . . . . . . . . . . . .. . . . . . . . . (XII`)
1
=i
ii l
a . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . .(XIII`)
These are the generalizations of formulae (XI), (XII) and (XIII).
Let us apply this to our example matrix (on page 21).
7/29/2019 Numerical Analysis 323
28/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/9
We get;
0 = 1 1 = 2 2 = -2 3 = -6 4 = -18 5 = 36
b1 = 2 b2 = 1 b3 = 5 b4 = -3 b5 = -1
c1 = -2 c2 = -2 c3 = 9 c4 = 3
a1 = -2 a3 = 1 a4 = -2 a5 = 1
We get 1/0 = 2 ; 2/1 = -1 ; 3/2 = 3 ; 4/3 = 3 ; 5/4 = -2
Thus from (XI`) we get
l1 = 2 u1 = 2
l2 = 1 u2 = -1
l3 = 3 u3 = 3
l4 = 3 u4 = 3
l5 = 2 u5 = -2
From (XII`) we get
22
2
1
11 =
==
u
Cw ; 2
1
2
2
22 =
==
u
Cw ;
333
9
3
33 ===
u
Cw ; 3
3
3
4
44 ===
u
Cw
From (XIII`) we get
22
2
1
22 =
==
l
a ; 1
1
1
2
33 ===
l
a ;
3
2
3
44
==
l
a ;
3
1
4
55 ==
l
a
Thus, we have LU decomposition,
7/29/2019 Numerical Analysis 323
29/161
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/10
=
=
20000
3
13000
03
2300
00110
00022
23000
033300
00320
00012
00002
13000
13900
02520
00112
00022
A
L U
in which the L and U have corresponding diagonal elements having the same magnitude.
7/29/2019 Numerical Analysis 323
30/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/1
DOOLITTLES LU DECOMPOSITION
We shall now consider the LU decomposition of a general matrix. The method we
describe is due to Doolittle.
Let A = (aij). We seek as in the case of a tridiagonal matrix, an LU decomposition inwhich the diagonal entries lii of L are all 1. Let L = (lii) ; U = (uij). Since L is a lower
triangular matrix, we have
lij = 0 if j > i ; and by our choice, lij =1.
Similarly, since U is an upper triangular matrix, we have
uij = 0 if i > j.
We determine L and U as follows : The 1st
row of U and 1st
column of L are determinedas follows :
=
=n
k
kkula1
1111
= l11 u11 Since l1k= 0 for k>1
= u11 Since l11 = 1.
.111 =u In general,
=
=n
k
kjkj ula1
11
= l11 u11 Since l1k= 0 for k>1
= u1j Since l11 = 1.
7/29/2019 Numerical Analysis 323
31/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/2
u1j = a1j . . . . . . . . . (I)Thus the first row of U is the same as the first row of A. The first column of L is
determined as follows:
=
=n
k
kjkj ula1
11
= lj1 u11 Since uk1 = 0 if k>1
lj1 = aj1/u11 . . . . . . . . . (II)Note : u11 is already obtained from (I).
Thus (I) and (II) determine respectively the first row of U and first column of L. The
other rows of U and columns of L are determined recursively as given below: Suppose
we have determined the first i-1 rows of U and the first i-1 columns of L. Now weproceed to describe how one then determines the i th row of U and ith column of L. Since
first i-1 rows of U have been determined, this means, ukj ; are all known for 1 k i-1 ; 1
j
n. Similarly, since first i-1 columns are known for L, this means, lik are all knownfor 1 i n ; 1 k i-1.
Now
=
=n
k
kjikij ula1
Since l==
i
kkjik ul1 ik= 0 for k>i
ijii
i
k
kjik ulul +=
=
1
1
7/29/2019 Numerical Analysis 323
32/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/3
since lij
i
k
kjik uul +=
=
1
1ii = 1.
=
=1
1
i
k
kjikijij ulau . . . . . ... . . . .(III)
Note that on the RHS we have a ij which is known from the given matrix. Also the sum
on the RHS involves lik for 1 k i-1 which are all known because they involve entriesin the first i-1 columns of L ; and they also involve ukj ; 1 k i-1 which are also known
since they involve only the entries in the first i-1 rows of U. Thus (III) determines the i
th
row of U in terms of the known given matrix and quantities determined upto the previous
stage. Now we describe how to get the ith column of L :
=
=n
k
kijkji ula1
Since u==i
k
kijk ul1
ki = 0 if k>i
iiji
i
k
kijk ulul +=
=
1
1
=
=
1
1
1 i
k
kijkji
ii
ji ulau
l..(IV)
Once again we note the RHS involves uii, which has been determined using (III); aij
which is from the given matrix; ljk; 1 k i-1 and hence only entries in the first i-1columns of L; and uki, 1 k i-1 and hence only entries in the first i-1 rows of U. ThusRHS in (IV) is completely known and hence lji, the entries in the i
th column of L are
completely determined by (IV).
7/29/2019 Numerical Analysis 323
33/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/4
Summarizing, Doolittles procedure is as follows:
lii = 1; 1st
row U = 1st
row of A ; Step 1 determining 1st
row of U and
lj1 = aj1/u11 1st
column of L.
For i 2; we determine
=
=1
1
i
k
kjikijij ulau ; j = i, i+1, i+2, .,n
(Note for j
7/29/2019 Numerical Analysis 323
34/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/5
1st row of U : Same as 1st row of A.
u11 = 2 ; u12 = 1 ; u13 = -1 ; u14 = 3
1st column of L:
l11 = 1;
l21 = a21/u11 = -2/2 = -1.
l31 = a31/u11 = 4/2 = 2.
l41 = a41/u11 = 6/2 = 3.
Second step:
2nd row of U : u12 = 0 (Because upper triangular)
u22 = a22 l21 u12 = 2 (-1) (1) = 3.
u23 = a23 l21 u13 = 6 (-1) (-1) = 5.
u24 = a24 l21 u14 = - 4 (-1) (3) = -1.
2nd
column of L : l12 = 0 (Because lower triangular)
l22 = 1.
l32 = (a32 l31 u12) /u22
= [14 (2)(1)]/3 = 4.
L42 = (a42 l41 u12) /u22
7/29/2019 Numerical Analysis 323
35/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/6
= [0 (3)(1)]/3 = -1.
Third Step:
3rd
row of U: u31 = 0 Because upper triangular
u32 = 0
u33 = a33 l31 u13 l32 u23
= 19 (2) (-1) (4)(5) = 1.
u34 = a34 l31 u14 l32 u24
= 4 (2) (3) (4)(-1) = 2.
3rd column of L : l13 = 0 Because lower triangular
l23 = 0
l33 =1
l43 = (a43 l41 u13 l42 u23)/ u33
= [-6 (3) (-1) (-1) (5)]/1
= 2.
Fourth Step:
4th row of U: u41 = 0
u42 = 0 Because upper triangular
u43 = 0
u44 = a44 l41 u14 l42 u24 l43 u34
= 12 (3) (3) (-1) (-1) (2) (2)
= -2.
7/29/2019 Numerical Analysis 323
36/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/7
4th column of L : l14 = 0 = l24 = l34 Because lower triangular
l44 = 1.
Thus.
=
1213
0142
0011
0001
L ; . . . . . . . . . . .(V)
=
2000
2100
1530
3112
U
and
A = LU.
This gives us the LU decomposition by Doolittles method for the given A.
As we observed in the case of the LU decomposition of a tridiagonal matrix; it isnot advisable to choose the lii as 1; but to choose in such a way that the diagonal entries
of L and the corresponding diagonal entries of U are of the same magnitude. We describe
this procedure as follows:
Once again 1st
row and 1st
column of U & L respectively is our first concern:
Step 1: a11 = l11 u11
Choose ( ) 1111111111 .sgn; aaual ==
Next 1011111
1 >====
forkaslulula kj
n
k
kjkij
7/29/2019 Numerical Analysis 323
37/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/8
11
1
l
au jij
Thus note that u1j have been scaled now as compared to what we did earlier.
Similarly,
11
11 u
al jj
These determine the first row of U and first column of L. Suppose we have determined
the first I-1 rows of U and first I-1 columns of L. We determine now the ith row of U and
ith
column of L as follows:
=
=n
k
kiikii ula1
for l=
=n
k
kiik ul1
ik= 0 if k>i
ii
i
k
iikiik ulul
=
+=1
1
saypulaul i
i
k
kiikiiiiii ,1
1
==
=
Choose ki
i
k
ikiiiii ulapl
=
==1
1
7/29/2019 Numerical Analysis 323
38/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/9
iiii ppu sgn=
iforklulula ikkj
i
k
ikkj
n
k
ikij >=== =
011
Q
ijiikj
i
k
ik ulul +=
=
1
1
=
=
kj
i
k
ikijij ulau1
1
lii
determining the ith row of U.
ki
n
k
jkji ula =
=1
iifkuul kiki
i
k
jk >== = 01Q
iijiki
i
k
jk ulul +=
=
1
1
=
=
1
1
i
k
kijkjiji ulal uii ,
thus determining the ith column of L.
Let us now apply this to matrix A in the example in page 30.
7/29/2019 Numerical Analysis 323
39/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/10
First Step:
2;22 1111111111 ==== ulaul
2
3;
2
1
2
1
11
1414
11
1313
11
1212 ======
l
au
l
au
l
au
2
3;
2
1;
2
1;2 14131211 ==== uuuu
22
2
11
21
21 ===u
al
222
4
11
3131 ===
u
al
232
6
11
4141 ===
ual
therefore
23
22
2
2
41
31
21
11
=
=
=
=
l
l
l
l
Second Step:
1221222222 ulaul =
7/29/2019 Numerical Analysis 323
40/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/11
( ) 32
122 =
=
3;3 2222 == ul
( )13212323 ulau = l22
( )3
53/2
126 =
=
[ ]14212424 ulau = l22
( ) ( )3
13/2
324 =
=
therefore
3
1;
3
5;3;0 24232221 ==== uuuu
( ) 2212313232 /uulal =
( ) 3/2
12214
=
34=
7/29/2019 Numerical Analysis 323
41/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/12
( ) 2212414242 /uulal =
( ) 3/2
1230
=
3=
therefore
3
34
3
0
42
32
22
12
=
=
=
=
l
l
l
l
Third Step:
23321331333333 ululaul =
( ) ( )
=
3
534
2
12219
= 1
1;1 3333 == ul
( ) 33243214313434 /lululau =
7/29/2019 Numerical Analysis 323
42/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/13
( ) ( ) 1/3
134
2
3224
=
= 2
2,1;0;0 34333231 === uuuu
[ ] 33234213414343 /uululal =
( ) ( ) 1/3
532
1236
=
= 2
therefore
=
=
=
=
2
1
0
0
43
33
23
13
l
l
l
l
Fourth Step:
344324421441444444 ulululaul =
( ) ( ) ( )( )223
13
2
32312
=
= -2
7/29/2019 Numerical Analysis 323
43/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/14
2;2 4444 == ul
2;0;0;0 44434241 ==== uuuu
=
=
=
=
2
0
0
0
44
34
24
14
l
l
l
l
Thus we get the LU decompositions,
=
22323
013422
0032
0002
L ;
=
2000
21003
1
3
530
2
3
2
1
2
12
U
in which iiii ul = , i.e. the corresponding diagonal entries of L and U have the same
magnitude.
Note: Compare this with the L and U of page 32. What is the difference.
The U in page 36 can be obtained from the U of page 32 by
(1)replacing the numbers in the diagonal of that U and keeping the same sign.Thus the first diagonal 2 is replaced by 2 ; 2nd diagonal 3 is replaced by 3 ,
third diagonal1 by 1 and 4th diagonal 2 by - 2 . These then give the diagonals
of the U in page 36.
(2)Divide each entry to the right of a diagonal in the U of page 32 by these replaceddiagonals.
7/29/2019 Numerical Analysis 323
44/161
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/15
Thus 1st row changes to 1st row of U in page 36
2nd row changes to 2nd row of U in page 36
3rd row changes to 3rd row of U in page 36
4th
row changes to 4th
row of U in page 36
This gives the U of page 36 from that of page 32.
The L in page 36 can be obtained from the L of page 32 as follows:
(1) Replace the diagonals in L by magnitude of the diagonals in U of page 36.(2) Multiply each entry below the diagonal of L by this new diagonal entry.
We get the L of page 32 changing to the L of page 36.
7/29/2019 Numerical Analysis 323
45/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/1
DOOLITTLES METHOD WITH ROW INTERCHANGES
We have seen that Doolittle factorization of a matrix A may fail the moment at stage i we
encounter a uii which is zero. This occurrence corresponds to the occurrence of zeropivot at the ith stage of simple Gaussian elimination method. Just as we avoided this
problem in the Gaussian elimination method by introducing partial pivoting we can adopt
this procedure in the modified Doolittles procedure. The Doolittles method which isused to factorize A as LU is used from the point of view of reducing the system
Ax = y
To two triangular systems
Lz = y
Ux = z
as already mentioned in page 17.
Thus instead of actually looking for a factorization A = LU we shall be looking for a
system,
A*x = y*
and for which A* has LU decomposition.
We illustrate this by the following example: The basic idea is at each stage calculate all
the uii that one can get by the permutation of rows of the matrix and choose that matrix
which gives the max. absolute value for uii.
As an example consider the system
Ax = y
where
=
3213
1451
3222
1213
A y =
1
3
8
3
We want LU decomposition for some matrix that is obtained from A by row interchanges.
We keep lii = 1.Stage 1:
1st diagonal of U. By Doolittle decomposition,
u11 = a11 = 3
7/29/2019 Numerical Analysis 323
46/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/2
If we interchange 2nd or 3rd or 4th rows with 1st row and then find the u11 for the new
matrix we get respectively u11 = 2 or 1 or 3. Thus interchange of rows does not give anyadvantage at this stage as we have already got 3 without row interchange for u11.
So we keep the matrix as it is and calculate 1st row of U, by Doolittles method.
.13
3;
3
1;
32;1
11
4141
11
3131
11
212111 ========
u
al
u
al
u
all
Thus
L is of the form
***1
01*3
1
0013
20001
; and
U is of the form ; A and Y remaining unchanged.
*000
**00
***0
1213
Stage 2
We now calculate the second diagonal of U: By Doolittles method we have
12212222 ulau = ( )3
81
3
22 =
=
Suppose we interchange 2nd row with 3rd row of A and calculate u22 : our new a22 is 5.
But note that the L gets in the 1st column 2nd and 3rd row interchanged. Therefore new l21
is1/3.
Suppose instead of above we interchange 2nd row with 4th row of A:
New a22 = 1 and new l21 = 1 and therefore new u22 = 1 (1) (1) = 0
Of these 14/3 has largest absolute value. So we prefer this. Therefore we interchange 2ndand 3rd row.
=
=
1
8
33
;
3213
3222
14511213
NewyNewA
7/29/2019 Numerical Analysis 323
47/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/3
=
=
*000
**00
**3
140
1213
;
1**1
01*3
2
0013
1
0001
NewUNewL
Now we do the Doolittle calculation for this new matrix to get 2nd row of U and 2nd
column of L.
13212323 ulau = ( ) ( )3
102
3
14 =
=
14212424 ulau = ( ) ( )3
21
3
11 =
=
2
nd
column of L:
[ ] 2212313232 uulal = ( ) ( )3
141
3
22
=
7
4=
[ ] 1112414242 uulal = ( )( )[ ]3
14113 = = 0
Therefore new L has form
1*01
017
4
3
2
001
3
10001
New U has form
*000
**003
2
3
10
3
140
1213
This completes the 2nd stage of our computation.
Note: We had three choices of u22to be calculated, namely 8/3, 14/3, 0 before we chose
14/3. It appears that we are doing more work than Doolittle. But this is not really so.For, observe, that the rejected u22 namely 8/3 and 0 when divided by the chosen u22
namely 14/3 give the entries of L below the second diagonal.
7/29/2019 Numerical Analysis 323
48/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/4
3rd Stage:
3rd diagonal of U:
233213313333 ululau = ( )
=
3
10
7
42
3
22
7
10=
Suppose we interchange 3rd row and 4th row of new A obtained in 2nd stage. We get new
a33 = 2.
But in L also the second column gets 3rd
and 4th
row interchanges
Therefore new l31 = 1 and new l32 = 0
Therefore new u33 = a33 l31 u13 l32 u23 ( )( ) ( )
+=
3
100212 = 4.
Of these two choices of u33 we have 4 has the larges magnitude. So we interchange 3rd
and 4th rows of the matrix of 2nd stage to get
=
=
8
1
3
3
3222
3213
1451
1213
NewYNewA
=
=
*000
*4003
2
3
10
3
140
1213
;
1*7
4
3
20101
0013
10001
NewUNewL
Now for this set up we calculate the 3rd stage entries as in Doolittles method:
243214313434 ululau = ( )( ) ( ) 43
20113 =
=
( ) 33234213414343 uululal =
7/29/2019 Numerical Analysis 323
49/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/5
( ) 43
10
7
42
3
22
= = 5/14.
=
=
*000
44003
2
3
10
3
140
1213
;
114
5
7
4
3
20101
0013
10001
NewUNewL
Note: The rejected u33 divided by chosen u33 gives l43.
4th Stage
[ ]3443244214414444 ulululau =
( ) ( )414
5
3
2
7
41
3
23
= = 13/7.
==
==
8
1
3
3
3222
3213
1451
1213
** YNewYANewA
New L = L* , New U = U*
,
7
13000
44003
2
3
10
3
140
1213
;
114
5
7
4
3
20101
0013
10001
**
=
= UL
and A* = L*U*
The given system Ax=y is equivalent to the system
A*x=y*
and hence can be split into the triangular systems,
L*z = y*
U*x = z
Now L*z = y* gives by forward substitution:
Z1 =3
7/29/2019 Numerical Analysis 323
50/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/6
21333
1221 ===+ zzz
411 1331 ===+ zzzz
814
5
7
4
3
24321 =++ zzzz
( ) ( ) ( ) 8414
52
7
43
3
24 =+
+
z
7
52
4 = z
=
7
524
23
z
Therefore U*x = z gives by back-substitution;
7
52
7
134 =x therefore x4 = -4.
311444 434343 ===+=+ xxxxxx
therefore x3 = 3
23
2
3
10
3
14432 = xxx
( ) ( ) 243
23
3
10
3
142 =
x
22 = x
323 4321 =+ xxxx
134623 11 =++ xx
Therefore the solution of the given system is
7/29/2019 Numerical Analysis 323
51/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/7
=
4
3
2
1
x
Some Remarks:
The factorization of a matrix A as the product of lower and upper triangular matrices is by
no means unique. In fact, the diagonal elements of one or the other factor can be chosen
arbitrarily; all the remaining elements of the upper and lower triangular matrices maythen be uniquely determined as in Doolittles method; which is the case when we choose
all the diagonal entries of L as 1. The name of Crout is often associated with triangular
decomposition methods, and in crouts method the diagonal elements of U are all chosenas unity. Apart from this, there is little distinction, as regards procedure or accuracy,
between the two methods.
As already mentioned, Wilkinsons suggestion is to get a LU decomposition in which
niul iiii = 1; .
We finally look at the cholesky decomposition for a symmetric matrix:
Let A be a symmetric matrix.
Let A = LU be a LU decomposition
Then A1
= U1
L1
U1
is also lower triangular
L1 is upper triangular
Therefore U1L1 is a decomposition of A1 as product of lower and upper triangular
matrices. But A1 = A since A is symmetric.
Therefore LU = U1L1
We ask the question whether we can choose L as U1; so that
A = U1U (or same as LL1)
Now therefore determining U automatically gets L = U1
We now do the Doolittle method for this. Note that it is enough to determine the rows ofU.
Stage 1: 1st row of U:
7/29/2019 Numerical Analysis 323
52/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/8
==
==n
k
kk
n
k
k uula1
12
1
1
111 11 kk ul =Q 1UL =Q
for k>1 since U is upper triangular112
u= 01 =kuQ
1111 au =
We finally look at the cholesky decomposition for a symmetric matrix:
Let A be a symmetric matrix.
Let A = LU be a LU decomposition
Then A1 = U1 L1 U1 is also lower triangular
L1 is upper triangular
Therefore U1L
1is a decomposition of A
1as product of lower and upper triangular
matrices. But A1
= A since A is symmetric.Therefore LU = U1L1
We ask the question whether we can choose L as U1; so that
A = U1U (or same as LL1)
Now therefore determining U automatically gets L = U1
We now do the Doolittle method for this. Note that it is enough to determine the rows ofU.
Stage 1: 1st row of U:
==
==n
k
kk
n
k
k uula1
12
1
1
111 11 kk ul =Q 1UL =Q
112
u= for k>1 since U is upper triangular.01 =kuQ
1111 au =
= =
==n
k
n
k
kikkiki uuula1 1
111
iuu 111= 101 >= forkukQ
1111 au =
1111 /uau ii = determines first row of U. and hence first column of L.
7/29/2019 Numerical Analysis 323
53/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/9
Having determined the 1st i-1 rows of U; we determine the ith row of U as follows:
= =
===n
k
n
k
kiikkikiikiiuluula
1 1
2Q
for k > i01
2 == =
ki
i
k
ki uu Q
ii
i
k
ki uu2
1
1
2 +=
=
=
=1
1
22i
k
kiiiii uau
=
=1
1
2i
k
kiiiii uau ; Note: uki are known for k i -1,
1st i-1 rows have already been obtained.
= =
==n
k
n
k
kjkikjikij uuula1 1
Now we need uij for j > i
==i
k
kjkiuu1
Because uki = 0 for k > i
=
+=1
1
i
k
ijiikjki uuuu
Therefore
ii
i
k
kjkiijij uuuau
=
=
1
1
7/29/2019 Numerical Analysis 323
54/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May
2004/10
=
=
=
=
ij
i
k
kjkiijij
i
k
kiiiii
uuuau
uau
1
1
1
1
2
determines the ith
row of U in terms of the previous rows. Thus we get U and L is U1.
This is called CHOLESKY decomposition.
Example:
Let
=
10131
1331
3351
1111
A
This is a symmetric matrix. Let us find the Cholesky decomposition.
1st row of U
==
==
==
==
1
1
1
1
111414
111313
111212
1111
uau
uau
uau
au
2nd row of U
7/29/2019 Numerical Analysis 323
55/161
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May
2004/11
( ) ( )( )( )
( ) ( )( )( )
===
===
===
22113
12113
215
2214122424
2213122323
122
2222
uuuau
uuuau
uau
3rd row of U
( ) ( )( ) ( )( )( )
===
===
2121111
1113
33242314133434
232
132
3333
uuuuuau
uuau
4th row of U
144110342
242
142
4444 === uuuau
=
1000
2100
2120
1111
U
==
1221
0111
0021
0001
1 LU and
A = LU
= LL1
= U1U
7/29/2019 Numerical Analysis 323
56/161
Numerical Analysis / Iterative methods for solving linear systems of equations Lecture notes
ITERATIVE METHODS FOR THE SOLUTION OF SYSTEMS EQUATION
In general an iterative scheme is as follows:
We have an nxn matrix M and we want to get the solution of the systems
x = Mx + y ..(1)
We obtain the solution x as the limit of a sequence of vectors, { }k
x which are obtained as
follows:
We start with any initial vector x(0)
, and calculate x(k)
from,
x(k) = Mx(k-1) + y .(2)
for k = 1,2,3, .. successively.
A necessary and sufficient condition for the sequence of vectors x(k) to converge to
solution x of (1) is that the spectral radius spM
of the iterating matrix M is less than 1 or
if1
7/29/2019 Numerical Analysis 323
57/161
Numerical Analysis / Iterative methods for solving linear systems of equations Lecture notes
)6......(....................
......00
...............
0...00
0......0
0......0
33
22
11
=
nna
a
a
a
D
the diagonal part of A; and
)7....(..............................
0
00
00
000
121
3231
21
=
nnn aaa
aa
a
L
K
KKKKK
K
KK
KK
the lower triangular part of A; and
)8(........................................
0...000
...............
...00
......0
223
112
= n
n
uu
uu
U
the upper triangular part of A.
Note that,
A = D + L + U (9).
We assume that ; i = 1, 2, , n (10)0iia
So that D-1
exists.
We now describe two important iterative schemes, below, for solving the system (3).
VittalRao/IISc, Bangalore M2/L1/V1/May 2004/2
7/29/2019 Numerical Analysis 323
58/161
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
JACOBI ITERATION
We write the system as in (4) as
11313212111 ..... yxaxaxaxa nn +=
22323121222 ..... yxaxaxaxa nn+=
. . . . . . . .(11)
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/1
...... ...... ......
nnnnnnnnn yxaxaxaxa += 112211 ..... We start with an initial vector,
( )
( )
( )
( )
)12........(....................
0
20
10
0
=
nx
x
x
xM
and substitute this vector for x on the RHS of (11) and calculate x1,x2, .., xn and this
vector is called x(1)
. We now substitute this vector on the RHS of (11) to calculate againx1, x2, .., xn and call this new vector as x
(2) and continue this procedure to calculate the
sequence x(k)
. We can describe this briefly as follows:
The equation (11) can be written as,
Dx = - (L + U) x + y . (13)
which we can write as
x = -D-1 (L+U) x +D-1 y,
giving
yJxx += (14)
where
J = -D-1
(L + U) .(15)
and, we get
x(0)
starting vector
,.......2,1;)1()( =+= kyJxx kk .(16)
as the iterating scheme. This is similar to (2) with the iterating matrix M as
J = -D-1 (L + U); J is called the Jacobi Iteration Matrix. The scheme will converge to the
solution x of our system if 11. Thus the Jacobi scheme for this system will not converge.
Thus, in example 3 we had a system for which the Jacobi scheme converged but Gauss
Seidel scheme did not converge; where in example 4 above we have a system for whichthe Jacobi scheme does not converge, but the Gauss Seidel scheme converges. Thus,
these two examples demonstrate that, in general, it is not correct to say that one scheme
is better than the other.
Let us now consider another example.
Example 5:
2x1 x2 =y1
-x1 + 2x2 x3 = y2
-x2 + 2x3 x4 =y3
-x3 + 2x4 = y4
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/8
7/29/2019 Numerical Analysis 323
73/161
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
Here
,
2100
1210
0121
0012
=A is a symmetric tridiagonal matrix.
The Jacobi matrix for this scheme is
=
0
2
100
2
10
2
10
02
10
2
1
002
10
J
The characteristic equation is,
164 - 122 + 1 = 0 (CJ)
Set 2 =
Therefore
162 - 12 + 1 = 0 (CJ1)
is the square root of the roots of (CJ1).
Thus the eigenvalues of J are 0.3090; 0.8090.
Hence
8090.0=sp
J ; and the Jacobi scheme will converge.
The Gauss Seidel matrix for the system is found as follows:
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/9
7/29/2019 Numerical Analysis 323
74/161
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
( )
=+
2100
0210
0021
0002
LD
=
0000
1000
0100
0010
U
( )
=+
2
1
4
1
8
1
16
1
021
41
81
002
1
4
1
0002
1
1LD
( )
=+=
0000
1000
0100
0010
21
41
81
161
02
1
4
1
8
1
002
1
4
1
0002
1
1ULDG
=
4
1
8
1
16
10
2
1
4
1
8
10
02
1
4
10
002
10
The characteristic equation of G is
0=GI , which becomes in this case
)(....................01216 234 GC=+
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/10
7/29/2019 Numerical Analysis 323
75/161
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
This can be factored as
( ) 011216 22 =+ Thus the eigenvalues of G are roots of
2 = 0 ; and
162 - 12 + 1 = 0 .(CG1)
Thus one of the eigenvalues of G is 0 (repeated twice), and two eigenvalues of G are
roots of (CG1). Notice that roots of (CG1) are same as those of (CJ1). Thus nonzero
eigenvalues of G are squares of eigenvalues of J. the nonzero eigenvalues of G are,
0.0955, 0.6545.
Thus,
16545.0
7/29/2019 Numerical Analysis 323
76/161
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/1
SUCCESSIVE OVERRELAXATION METHOD (SOR METHOD)
We shall now consider SOR method for the system
Ax = y ..(I)
We take a parameter 0 and multiply both sides of (I) to get an equivalent system,
Ax = y (II)
Now
( )ULDA ++= We write (II) as
(D + L + U)x = y,
i.e.
(D + L) = - Ux + y
i.e.
(D + L)x + (-1) Dx = - Ux +y
i.e.
(D + L)x = - [( 1)D + U]x + y
i.e.
x = - (D + L)-1 [(-1)D + U]x + [D + L]-1y.
We thus get the SOR scheme as
( ) ( )
( ) ;
0
1
=
+=+
x
yxMx kk
initial guess (III)
where,
( ) ( )[ ]uDLDM ++=
11
and
( ) yLDy 1 +=
M is the SOR matrix for the system.
7/29/2019 Numerical Analysis 323
77/161
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/2
Notice that if = 1 we get the Gauss Seidel scheme. The strategy is to choose suchthat ,1 0). Usually is chosen
between 1 and 2. Of course, one must analysesp
M as a function of and find that
value 0 of for which this is minimum and work with this value of0.
Let us consider an example of this aspect.
Example 6:
Consider the system given in example 5.
For that system,
M = - (D + L)-1
[(-1) D +U]
+++
++
+
=
23243243
23232
22
4
11
8
1
2
1
2
1
16
1
4
1
4
1
8
1
8
12
1
4
11
8
1
2
1
2
1
4
1
4
1
02
1
4
11
2
1
2
1
002
11
and the characteristic equation is
( ) ( ) ( ) MC.................011211624224 =+++
Thus the eigenvalues of M are roots of the above equation. Now when is = 0 a root?If = 0 we get from (CM),
16(-1)4 = 0 = 1,
i.e. in the Gauss Seidel case. So let us take 1; so = 0 is not a root. So we candivide the above equation (CM) by
42 to get
( ) ( )01
112
116 2
22
2
2
=++
+
Setting
7/29/2019 Numerical Analysis 323
78/161
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/3
( )
2
2
2 1+= we get
011216 24 =+
which is the same as (CJ). Thus
.8090.0;3090.0 =
Now
( ) 22
21
=
+= 0.0955 or 0.6545 .(*)
Thus, this can be simplified as
( ) ( )2
1
22221
4
11
2
1
=
as the eigenvalues of M.
With = 1.2 and using the two values of2 in (*) we get,
= 0.4545, 0.0880, -0.1312 i (0.1509).
as the eigenvalues. The modulus of the complex roots is 0.2
Thus
spM when = 1.2 is 0.4545
which is less thatsp
J = 0.8090 andsp
G = 0.6545 computed in Examples. Thus for
this system, SOR with = 1.2 is faster than Jacobi and Gauss Seidel scheme.
We can show that in this example when = 0 = 1.2596, the spectral radius 0M is
smaller than M for any other . We have
2596.1M = 0.2596
Thus the SOR scheme with = 1.2596 will be the method which converges fastest.
Note:
We hadsp
M 2.1 = 0.4545
7/29/2019 Numerical Analysis 323
79/161
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/4
And
spM 2596.1 = 0.2596
Thus a small change in the value of brings about a significant change in the spectral
radiussp
M .
7/29/2019 Numerical Analysis 323
80/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/1
EIGENVALUES AND EIGENVECTORS
Let A be an nxn matrix. A scalar is called an eigenvalue of A if there exists a nonzeronx1 vector x such that
Ax = xExample:
Let
=
7816
438
449
A
1=Let Consider
=
0
2
1
x. We have
=
=
=
0
2
1
1
0
2
1
0
2
1
7816
438
449
Ax
( ) xx == 1
1= is such that there exists a nonzero vector x such that Ax = x. Thus is aneigenvalue of A.
Similarly, if we take = 3, we find that
=
0
2
1
x
Ax = x. Thus, = 3 is also an eigenvalue of A.
Let be an eigenvalue of A. Then any nonzero x such that Ax = x is called aneigenvector of A.
Let be an eigenvalue of A. Let,
.: xAxCx n ==
Then : (i) is nonempty. = nxQ
7/29/2019 Numerical Analysis 323
81/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/2
+
+=+==
yx
yxyxAyAyxAxyxii )()(,,)(
(iii)
For any constant )(; xxAx == )()( xxA =
x
Thus is a subspace of Cn. This is called the characteristic subspace or the
eigensubspace corresponding to the eigenvalue .
Example:
Consider the A in the example on page 1. We have sum = -1 is an eigenvalue. What is-1, the eigensubspace corresponding to 1?
We want to find all x such that
Ax = -x
i.e., (A+I)x = .
i.e., we want to find all solutions of the homogeneous system Mx = ; where
=+=
7816
448
448
IAM
We now can use our row reduction to find the general solution of the system.
000
0002
1
2
11
000
000
4481
12
13
8
1
2
RRR
RR
M
Thus, 32121
21 xxx +=
Thus the general solution of (A+I) x = is
7/29/2019 Numerical Analysis 323
82/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/3
+
=
+
2
0
1
2
1
0
2
1
2
12
1
2
1
32
3
2
32
xx
x
x
xx
+
=
2
0
1
0
2
1
21 AA
where A1 and A2 are arbitrary constants.
Thus -1 consists of all vectors of the form
+
20
1
02
1
21 AA .
Note: The vectors form a basis for
2
0
1
,
0
2
1
-1 and therefore
dim -1 = 2.
What is 3 the eigensubspace corresponding to the eigenvalue 3 for the above matrix
We need to find all solutions of Ax = 3x,
i.e., Ax 3x =
i.e., Nx =
Where
==
4816
408
4412
3IAN
Again we use row reduction
7/29/2019 Numerical Analysis 323
83/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/4
+
000 34
3
80
4412
3
4
3
80
3
4
3
80
4412
43
12
13
3
2
3
4
RR
RR
RR
N
321 4412 xxx +=
323
4
3
8xx = 23 2xx =
2221 128412 xxxx =+=
12 xx =
12312 22; xxxxx === The general solution is
=
2
1
1
2
1
1
1
1
x
x
x
x
Thus 3 consists of all vectors of the form
2
1
1
Where is an arbitrary constant.
Note: The vector forms a basis for
2
1
1
3 and hence
dim. 3 = 1.
Now When can a scalar be an eigenvalue of a matrix A? We shall now investigate thisquestion. Suppose is an eigenvalue of A.
7/29/2019 Numerical Analysis 323
84/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/5
This There is a nonzero vector x such that Ax = x.
.;)( = andxxIA
The system = xIA )( has at least one nonzerosolution.
nullity (A - I) 1
rank(A - I) < n
(A - I) is singular
det. (A - I) = 0
Thus, is an eigenvalue of A det. (A - I) = 0.
Conversely, is a scalar such that det. (A - I) = 0.
This (A - I) is singular
rank(A - I) < n
nullity (A - I) 1
The system = xIA )( has nonzero solution. is an eigenvalue of A.
Thus, is a scalar such that det. (A - I) = 0 is an eigenvalue.
Combining the two we get,
is an eigenvalue of A
det. (A - I) = 0
det. (I - A) = 0
Now let C() = det. (I - A)
Thus we see that,
The eigenvalues of a matrix A are precisely the roots of C() = det. (I - A).
7/29/2019 Numerical Analysis 323
85/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/6
( )
nnnn
n
n
aaa
aaa
aaa
C
=
K
KKKK
KKKK
K
K
21
22221
11211
( ) ( ) Aaa nnnnn
.det1111 ++++=KK
Thus ; C() is a polynomial of degree n. Note the leading coefficient of C() is 1. Wesay C() is a monic polynomial of degree n. This is called CHARACTERISTICPOLYNOMIAL of A. The roots of the characteristic polynomial are the eigenvalues of
A. The equation C() = 0 is called the characteristic equation.
Sum of the roots of C() = Sum of the eigenvalues of A = a11 + . . . . . . + ann ,
and this is called the TRACE of A.
Product of the roots of C() = Product of the eigenvalues of A = det. A.
In our example in page 1 we have
=7816
438
449
A
( )7816
438
449
).(det
+
==
AIC
781
431
441
321
+
+
+
++
CCC
7/29/2019 Numerical Analysis 323
86/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/7
( )781
431
441
1
+=
340
010
441
)1(12
13
+
+=
RR
RR
( )( )( )311 ++=
( ) ( )31 2 += Thus the characteristic polynomial is
( ) ( )31)( 2 += C The eigenvalues are 1 (repeated twice) and 3.
Sum of eigenvalues = (-1) + (-1) + 3 = 1
Trace A = Sum of diagonal entries.
Product of eigenvalues = (-1) (-1) (3) = 3 = det. A.
Thus, if A is an nxn matrix, we define the CHARACTERISTIC POLYNOMIAL as,
AIC = )( . . . . . . . . . . . . .(1)
and observe that this is a monic polynomial of degree n. When we factorize this as,
( ) ( ) ( ) kakaa
C = KK21 21)( . . . . . . . .(2)Where 1, 2, . . . . . ., k are the distinct roots; these distinct roots are the distincteigenvalues of A and the multiplicities of these roots are called the algebraic
multiplicities of these eigenvalues of A. Thus when C() is as in (2), the distincteigenvalues are 1, 2, . . . . . ., kand the algebraic multiplicities of these eigenvalues are
respectively, a1, a2, . . . . . , ak.For the matrix in Example in page 1 we have found the characteristic polynomial on page
6 as
( ) ( 31)( 2 += C )
7/29/2019 Numerical Analysis 323
87/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/8
Thus the distinct eigenvalues of this matrix are 1 = -1 ; and 2 = 3 and their algebraicmultiplicities are respectively a1 = 2 ; a2 = 1.
Ifi is an eigenvalues of A the characteristic subspace corresponding to i is i and is
defined as
{ }xAxx ii == :
The dimension of i is called the GEOMETRIC MULTIPLICITY of the eigenvalue i
and is denoted by gi.
Again for the matrix on page 1, we have found on pages 3 and 4 respectively that, dim -1 = 2 ; and dim. 3 = 1. Thus the geometric multiplicities of the eigenvalues 1 = -1 and2 = 3 are respectively g1 = 2 ; g2 = 1. Notice that in this example a1 = g1 = 2 ; and a2 = g2
= 1. In general this may not be so. It can be shown that for any matrix A having C() asin (2),
1 gi ai ; 1 i k . . . . . . . . . . . .(3)
i.e., for any eigenvalue of A,
1 geometric multiplicity algebraic multiplicity.
We shall study the properties of the eigenvalues and eigenvectors of a matrix. We shall
start with a preliminary remark on Lagrange Interpolation polynomials :
Let 1, 2, . . . . . . . ., s be a distinct scalars, (i.e., ij if i j ). Consider,
)())(())((
)())(())(()(
1121
1121
siiiiiii
sii
ip
=
+
+
KK
KK
)(
)(
1ji
j
sjij
=
for i = 1,2, . . . . . . ., s . . . . . . .. (4)
Then pi() are all polynomials of degree s-1.
Further notice that ( ) ( ) ( ) 0)(111 ====== + siiiiii pppp KK
( ) 1=iip
7/29/2019 Numerical Analysis 323
88/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/9
Thus pi()are all polynomials of degree s-1 such that,
ijjip = if j i . . . . . . . . . . (5)
We call these the Lagrange Interpolation polynomials. If p() is any polynomial ofdegree s-1 then it can be written as a linear combination ofp1(),p2(), . . ., ps()as follows:
( ) ( ) ( ) ( ) ( ) ( ) ( ) ss ppppppp +++= L2211 . . . . (6)
( ) (=
=s
i
ii pp1
)
With this preliminary, we now proceed to study the properties of the eigenvalues and
eigenvectors of an nxn matrix A.
Let 1, . . . . , k be the distinct eigenvalues of A. Let 1, 2, . . . , k be eigenvectorscorresponding to these eigenvalues respectively ; i.e., i are nonzero vectors such that
Ai = ii . . . . . . . . . . . .(6)From (6) it follows that
iiiiiiii AAAAA 22
)()( ====
iiiiiiii AAAAA 32223 )()( ====
and by induction we get
iim
i
mA = for any integer m 0 . . . . . . . . . . .(7)(We interpret A0 as I).
Now let,s
saaap +++= KK10)(
be any polynomial. We define p(A) as the matrix,
s
s AaAaIaAp +++= KK10)(
7/29/2019 Numerical Analysis 323
89/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/10
Now
i
s
si AaAaIaAp )()( 10 +++= KK
issii AaAaa +++= KK10
by (6)iis
siii aaa +++= KK10
iis
si aaa )( 10 +++= KK
.)( iip = Thus,
Now are the eigenvectors, 1, 2, . . . . , kcorresponding to the distinct eigenvalues1, 2, . . . . , kof A, linearly independent ?
Ifi is any eigenvalue of A and i is an eigenvector corresponding to i then
for any polynomial p() we have .)()( iii pAp =
In order to establish this linear independence, we must show that
0212211 =====+++ KnKK CCCCCC KK . . . (8)Now if in (4) & (5) we take s = k ; i = i then we get the Lagrange Interpolationpolynomials as
( ))(
)(
1ji
j
kji
ij
p
=
; i = 1,2,.., k (9)
and
ijjip = if j i (10)Now,
nkkCCC =+++ ....2211
For 1 i k,
7/29/2019 Numerical Analysis 323
90/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/11
( )[ ] ( ) nnikki ApCCCAp ==+++ ....2211
( ) ( ) nkikii ApCApCApC =+++ ....)( 2211
( ) ( ) ,....)( 222111 nkkikii pCpCpC =+++ (by property I on page 10)
;1; kiC ii = by (10)
kiCi = 1;0 since i are nonzero vectors
Thus
0........ 212211 =====+++ nnkk CCCCCC proving (8). Thus we have
Eigen vectors corresponding to distinct eigenvalues of A are linearlyindependent.
7/29/2019 Numerical Analysis 323
91/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/1
SIMILAR MATRICES
We shall now introduce the idea of similar matrices and study the properties of similar
matrices.
DEFINITION
An nxn matrix A is said to be similar to a nxn matrix B if there exists a nonsingular nxnmatrix P such that,
P-1 A P = B
We then write,
A B
Properties of Similar Matrices
(1) Since I-1
A I = A it follows that A A
(2) A B P, nonsingular show that., P-1 A P = B
A = P B P-1
A = Q-1 B P, where Q = P-1 is nonsingular
nonsingular Q show that Q-1 B Q = A
B A
Thus
A B B A
(3) Similarly, we can show that
A B, B C A C.
(4) Properties (1), (2) and (3) above show that similarity is an equivalence relation on theset of all nxn matrices.
(5) Let A and B be similar matrices. Then there exists a nonsingular matrix P such that
A = P-1 B P
Now, let CA() and CB () be the characteristic polynomials of A and B respectively. Wehave,
( ) BPPIAICA1==
BPPPP 11 =
( )PBIP = 1
PBIP = 1
7/29/2019 Numerical Analysis 323
92/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/2
1sin 1 == PPceBI
= CB ()
Thus SIMILAR MATRICES HAVE THE SAME CHARACTERISTIC
POLYNOMIALS .
(6) Let A and B be similar matrices. Then there exists a nonsingular matrix P such that
A = P-1 B P
Now for any positive integer k, we have
( )( ) ( )4444 34444 21
ktimes
kBPPBPPBPPA
111 ..... =
= P-1
Bk
P
Therefore, Ak
= On P-1
Bk
P = On
Bk= On
Thus if A and B are similar matrices then Ak= On Bk= On .
Now let p() = a0 + a1 + .. + ak be any polynomial.
Then
( ) kkAaAaIaAp +++= .....10
PBPaPBPaBPPaIak
k
121
2
1
10 ..... ++++=
PBaBaBaIaP kk++++=
.....22101
( )PBpP 1=
Thus
( ) ( ) nn OPBpPOAp ==1
( ) nOBp =
Thus IF A and B ARE SIMILAR MATRICES THEN FOR ANY POLYNOMIAL p ();p (A) = On p (B) = On .
(7) Let A be any matrix. By A(A) we denote the set of all polynomials p() such that
p(A) = On, i.e.
7/29/2019 Numerical Analysis 323
93/161
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/3
A (A) = {p() : p(A) = On}
Now from (6) it follows that,
IF A AND B ARE SIMILAR MATRICES THEN A(A) = A (B) .
Then set A (A) is called the set ANNIHILATING POLYNOMIALS OF A