Embed Size (px)
Citation preview
– LU Decomposition– Matrix Inverse– System Condition– Special Matrices– Gauss-Seidel
Part 4b: NUMERICAL LINEAR ALGEBRA
In Gauss elimination, both coefficieints and constants are munipulated until an upper-triangular matrix is obtained.
In some applications, the coefficient matrix [A] stays constant while the right-hand-side constants vector (b) changes.
[L][U] decomposition does not require repeated eliminations. Once [L][U] decomposition is applied to matrix [A], it can be repeteadly used for different values of (b) vector.
LU Decomposition:
...
11313212111 ... bxaxaxaxa nn '2
'23
'232
'22 ... bxaxaxa nn
''3
''33
''33 ... bxaxa nn
)1()1( nnn
nnn bxa
Decomposition methodology:
bxA
Solution to the linear system
The system can also be stated in an upper triangular form:
0 bxAor
dxU or 0 dxU
Now, suppose there exist a lower triangular matrix (L) such that
bxAdxUL
Then, it follows that
AUL and bdL
Solution for (x) can be obtained by a two-step strategy (explained next).
Decomposition strategy:
bxA
U L
Decomposition
)()( bdL Apply forward substitution to calculate (d)
)()( dxU Apply backward substitution to calculate (x)
The process involves one decomposition, one forward substitution, and one backward substitution processes.
Once matrices L and U are computed once; manipulated constant vector (d) is repeatedly calculated from matrix L; hence vector (x).
LU Decomposition and Gauss Elimination:Gauss elimination processes involves an LU decomposition in itself.Forward elimination produces an upper triangular matrix:
In fact, while U is formed during elimination, an L matrix is formed such that (for 3x3)
..00
....0
......
U
1
01
001
3231
21
ff
fL
where
11
2121 a
af
11
3131 a
af
22
'32
32 a
af …
ULA This decomposition is unique when the diagonals of L are ones.
EX 10.1 : Apply LU decomposition based on the Gauss elimination for Example 9.5 (using 6 S.D.):
Coefficient matrix:
Forward elimination resulted in the following upper triangular form:
Lower triangular matrix will have
102.03.0
3.071.0
2.01.03
A
0120.1000
293333.000333.70
2.01.03
U
10271300.0100000.0
010333333.0
001
1
01
001
1
01
001
'22
'32
11
31
11
21
3231
21
a
a
a
aa
a
ff
fL
Check the result:
We obtain: compare to:
0120.1000
293333.000333.70
2.01.03
10271300.0100000.0
010333333.0
001
ULA
99996.92.03.0
3.070999999.0
2.01.03
A
102.03.0
3.071.0
2.01.03
A
Some round-off error is introduced
To find the solution:
®Calculate (d) by applying one forward substitution.
® Calculate (x) by applying one back substitution.
)()( bdL
)()( dxU
[L] facilitates obtaining modified (b) each time (b) has been changed during calculations.
EX 10.2: Solve the system in the previous example using LU decomposition:
We have:
> Apply the forward substitution:
> Apply the backward substitution:
4.71
3.19
85.7
10271300.0100000.0
010333333.0
001
3
2
1
d
d
d
0843.70
5617.19
85.7
3
2
1
d
d
d
4.71
3.19
85.7
0120.1000
293333.000333.70
2.01.03
3
2
1
x
x
x
00003.7
5.2
3
3
2
1
x
x
x
0120.1000
293333.000333.70
2.01.03
10271300.0100000.0
010333333.0
001
ULA
1
..
1
ULA
Total FLOPs with LU decomposition 23
3nO
n same as Gauss
elimination
Crout Decomposition:
(Doolittle decomposition/factorizaton)
1
..
1
ULA (Crout decomposition)
They have comperable performances.Crout decompositon can be implemented by a
concise series of formulas. (see the book). Storage can be minimized:
> No need to store 1’s in U.> No need to store 0’s in L and U.> Elements of U can be stored in zeros of L.
row operation
Colu
mn
ro o
pera
tion
L (forming)
A(remaining)
U (forming)
Matrix Inverse
IAAAA 11
If A is a square matrix,there exist an A-1,s.t.
LU decomposition offers an efficient way to find A-1.
bxA
U L
)()( ,: jIdL
)()( 1,: dAU j
For constant vector, enter (I:,i ) (ith column of the identity matrix.)
Solution gives ith column of A-1.
forward substitution
Backward substitution
decomposition
EX 10.3 : Use LU decomposition to determine the inverse of the system in EX 10.1
Corresponding upper and lower triangular matrices are
To calculate the first column of A-1 :
> Forward substitution:
102.03.0
3.071.0
2.01.03
A
0120.1000
293333.000333.70
2.01.03
U
10271300.0100000.0
010333333.0
001
L
0
0
1
10271300.0100000.0
010333333.0
001
3
2
1
d
d
d
1009.0
03333.0
1
3
2
1
d
d
d
> Back substitution:
To calculate the second column To calculate the third column
We finally get
1009.0
03333.0
1
0120.1000
293333.000333.70
2.01.03
3
2
1
x
x
x
01008.0
00518.0
33249.0
3
2
1
x
x
x First column of A-1
0
1
0
3
2
1
b
b
b
00271.0
142903.0
004944.0
3
2
1
x
x
x
1
0
0
3
2
1
b
b
b
09988.0
004183.0
006798.0
3
2
1
x
x
x
09988.000271.001008.0
004183.0142903.000518.0
006798.0004944.033249.01A
Many engineering problems can be represented by a linear equation
The formal solution to this equation
Importance of Inverse in Engineering Applications:
bxA
Response (e.g., deformation)
Stimulus (e.g., force)
System design matrix
bAx 1
For a 3x3 system we can write explicitly
31132
1121
1111 bababax
31232
1221
1212 bababax
31332
1321
1313 bababax
There is a linear relationship between stimulus and response. Proportionality constants are the coefficients of A-1 .
System ConditionCondition number indicates ill-conditioning of a system.We will determine condition number using matrix norms.
Matrix norms:A norm is a measure of the size of a multi-component entity
(e.g., a vector)
2/1
1
2
2
n
iiexxx 2-norm
(Euclidean norm)
n
iixx
11 1-norm
pn
i
p
ipxx
/1
1
p-norm
1x
2x
3x
2x
We can extend Euclidean norm for matrices:
There are other norms too…, e.g.,
Each of these norms returns a single (positive) value for the characteristics of the matrix.
2/1
1 1
2
,
n
i
n
jjie
aA (Frobenius norm)
n
jij
niaA
11max (row-sum norm)
n
iij
njaA
11max (column-sum norm)
( )
Matrix condition number can be defined as
If Cond [A] >> 1 ill-conditioned matrix It can be shown that
For example;[A] contains element of t S.F. (precision of 10-t)
Cond [A] 10c
Matrix Condition Number:
1 AAACond
A
AACond
x
x
i.e., the relative error of the computed solution cannot be larger than the relative error of the coefficients of [A] multiplied by the condition number.
1ACond
(x) will contain elements of (t-c) S.F.(precision of 10c-t)
EX 10.4 : Estimate the condition number of the 3x3 Hilbert matrix using row sum norm
First normalize the matrix by dividing each row by the largest coefficient:
Row-sum norm:
5/14/13/1
4/13/12/1
3/12/11
AHilbert matrix is inherently ill-conditioned.
5/34/31
2/13/21
3/12/11
A
833.13/12/11
1667.22/13/21
35.25/34/31
5/34/31
2/13/21
3/12/11
A 35.2
A
Inverse of the scaled matrix:
Row-sum norm:
Condition number:
matrix is ill-conditioned.
e.g., for a single precision (7.2 digits) computation;
609030
609636
101891A
609030
609636
101891A 1926096361
A
2.451)192)(35.2( ACond
65.2)2.451log( c (7.2-2.65)=4.55 ~ 4 S.F. in the solution! (precision of ~10-4)
this part takes the longest time of computation.
Iterative refinement:This technique especially useful for reducing round-off errors.Consider a system:
Assume an approximate solution of the form
We can write a relationship between exact and approximate solutions:
1313212111 bxaxaxa
2323222121 bxaxaxa
3333232131 bxaxaxa
oooo bxaxaxa 1313212111 oooo bxaxaxa 2323222121 oooo bxaxaxa 3333232131
111 xxx o 222 xxx o 333 xxx o
Insert these into the original equations:
Now subtract the approximate solution from above to get
This a new set of simultaneous linear equation which can be solved for the correction factors.
Solution can be improved by applying the corrections to the previous solution (iterative refinement procedure)
It is especially suitable for LU decomposition since constant vector (b) continuously changes.
1331322121111 )()()( bxxaxxaxxa
2332322221121 )()()( bxxaxxaxxa
3333322321131 )()()( bxxaxxaxxa
111313212111 ebbxaxaxa o
222323222121 ebbxaxaxa o
333333232131 ebbxaxaxa o
Special Matrices
In engineering applications, special matrices are very common.> Banded matrices
BW=3 tridiagonal system > Symmetric matrices
> Spare matrices (most elements are zero)
0ija 2/)1( BWjiBW
if
jiij aa or TAA
only black areas are non-zero
Application of elimination methods to spare matrices are not efficient (e.g., need to deal with many zeros unnecessarily).
We employ special methods in working with these systems.
Cholesky Decomposition:This method is applicable to symmetric matrices.
A symmetric matrix can be decomposed as
TLLA
Symmetric matrices are very common in engineering applications. So, this method has wide applications.
or ii
i
jkjijki
ki l
lla
l
1
1 1,...,2,1 kifor
1
1
2k
jkjkkkk lal
EX 11.2 : Apply Cholesky decomposition to
Apply the recursion relation:
97922555
2255515
55156
A
4495.261111 al
1237.64495.2
15
11
2121
l
al 1833.4)1237.6(55 22
212222 lal
454.224495.2
55
11
3131
l
al 916.20
1833.4
)454.22(1237.6225
22
31213232
l
llal
1106.6
)916.20()454.22(979 22
232
2313333
llal
1106.6916.20454.22
1833.41237.6
4495.2
L
k=1
(k=2,i=1)
(k=3, i=1) (k=3, i=2)
Gauss-Seidel Iterative methods are strong alternatives to elimination methods
In iterative methods solution is constantly improved so there is no concern of round-off errors.
As we did in root finding, > start with an initial guess. > iterate for refined estimates of the solution.
Gauss-Seidel is one of the most commonly used iterative method.
For the solution of
We write each unknown in the diagonal in terms of the other unknowns:
)()( bxA
In case of a 3x3 system:
11
31321211 a
xaxabx
11
32312122 a
xaxabx
33
23213133 a
xaxabx
Start with initial guesses x2 and x3 calculate new x1
Use new x1 and old x3 calculate new x2
Use new x1 and x2 calculate new x3
In Gauss Seidel, new estimates are immediately used in subsequent calculations.
Alternatively, old values (x1 , x2 , x3) are collectively used to calculate new values (x1 , x2 , x3) Jacobi iteration (not commonly used)
iterate...
EX 11.2 : Use Gauss-Seidel method to obtain the solution of
Gauss-Seidel iteration:
Assume initial guesses all are 0.
85.72.01.03 321 xxx
3.193.071.0 321 xxx
4.71102.03.0 321 xxx
true resultsx1 = 3x2 = - 2.5x3 = 7
3
2.01.085.7 321
xxx
7
3.01.03.19 312
xxx
10
2.03.04.71 213
xxx
616667.23
0085.71
x 794524.2
7
0)616667.2(1.03.192
x
005610.710
)794524.2(2.0)616667.2(3.04.713
x
For the second iteration, we repeat the process:
The solution is rapidly converging to the true solution.
990557.23
)005610.7(2.0)794524.2(1.085.71
x
499625.27
)005610.7(3.0)990557.2(1.03.192
x
000291.710
)499625.2(2.0)990557.2(3.04.713
x
%31.0t
%015.0t
%0042.0t
Convergence in Gauss-Seidel:
Gauss-Seidel is similar to the fixed-point iteration method in root finding methods. As in the fixed-point iteration, Gauss-Seidel also is prone to
> divergence > slow convergence
Convergence of the method can be checked by the following criteria:
Fortunately, many engineering applications fulfill this requirement.
n
ijj
ijii aa1
that is, the absolute value of the diagonal coefficient in each of the row must be larger than sum of the absolute values of all other coefficients in the same row.(diagonally dominant system)
Improvement of convergence by relaxation:
After each value of x is computed using Gauss-Seidel equations, the value is modified by a weighted average of the old and new values.
If 0<<1 underrelaxation (to make a system converge) If 1<<2 overrelaxation (to accelerate the convergence )
The choice of is empirical and depends on the problem.
oldi
newi
newi xxx )1( 20
