MA/CS 375 Fall 2003 1
MA/CS 375
Fall 2003
Lecture 19
MA/CS 375 Fall 2003 2
Matrices and Systems
MA/CS 375 Fall 2003 3
Upper Triangular Systems
11 1 12 2 1 1
1 22 2 2 2
1 2
...
0 ...
0 0 ...
N N
N N
NN N N
x x x b
x x x b
x x x b
U U U
U U
U
MA/CS 375 Fall 2003 4
Upper Triangular Systems
Volunteer: tell us how to solve this? (hint: this is not difficult)
11 1 12 2 1 1
1 22 2 2 2
1 2
...
0 ...
0 0 ...
N N
N N
NN N N
x x x b
x x x b
x x x b
U U U
U U
U
MA/CS 375 Fall 2003 5
Example: Upper Triangular Systems
11 1 12 2 13 3 1
1 22 2 23 3 2
1 2 33 3 3
0
0 0
x x x b
x x x b
x x x b
U U U
U U
U
MA/CS 375 Fall 2003 6
Upper Triangular Systems1) Solve the Nth equation
2) Use the value from step 1 to solve the (N-1)th equation
3) Use the values from steps 1 and 2 to solve the (N-2)th equation
4) keep going until you solve the whole system.
MA/CS 375 Fall 2003 7
Backwards Substitution
1 1 1,1, 1
2 2 2, 1 1 2,2, 2
1
1 1 1,21,1
1
1
1
1
1
N NNN
N N N N NN N
N N N N N N N NN N
j N
i i ij jj iii
j N
j jj
x b
x b x
x b x x
x b x
x b x
U
UU
U UU
UU
UU
MA/CS 375 Fall 2003 8
Matlab Code for Backward
Substitution
1
1 j N
i i ij jj iii
x b x
U
U
MA/CS 375 Fall 2003 9
Testing Backwards Substitution
MA/CS 375 Fall 2003 10
Finite Precision Effects
• Notice that x(1) depends on the values of x(2),x(3),…,x(N)
• Remembering that round off rears its ugly head when we deal with (small+large) numbers
• Volunteer to design a U matrix and b vector, which do not give good answers when used in this algorithm.
• (goal is to give you some intuition for cases which will break an algorithm)
MA/CS 375 Fall 2003 11
Team Exercise
• Who can build a “reasonable”, non-singular, U matrix and b vector which has the worst answer for U\b
• 10 Minutes only
MA/CS 375 Fall 2003 12
LU Factorization• By now you should be persuaded that it is not difficult to solve a problem of the form Lx=b or Ux=b using forward/backward elimination
• Suppose we are given a matrix A and we are able to find a lower triangular matrix L and an upper triangular matrix U such that:
A = LU
• Then to solve Ax=b we can solve two simple problems:
• Ly = b• Ux = y • (sanity check LUx = L(Ux) = Ly =b)
MA/CS 375 Fall 2003 13
Example LU Factorization
• Suppose we are given:
• Then we can write A = LU where:
• Let’s check that:
2 3
1 2
A
1 0
0.5 1
L
2 3
0 0.5
U
1 0 2 3 1*2 0*0 1*3 0*0.5 2 3
0.5 1 0 0.5 0.5*2 1*0 0.5*3 1*0.5 1 2
LU
MA/CS 375 Fall 2003 14
LU Factorization (in words)• Construct a sequence of multiplier matrices (M1, M2, M3, M4,…., MN-1) such that:
(MN-1…. M4 M3 M2 M1)A is upper triangle.
•A suitable candidate for M1 is the matrix ( 4 by 4 case):
13
4
1 0 0 0
1 0 0
0 1 0
0 0 1
21
11
1
11
1
11
A
A
M A
A
A
A
MA/CS 375 Fall 2003 15
Check to See What M1 Does to A
11 12 13 14
21 22 23 2413
31 32 33 34
41 42 43 444
11 12 13 14
21 1321 12 21 1422 23 24
11 11 11
31 12 31 1332 33
11 11
1 0 0 0
1 0 0
0 1 0
0 0 1
0
0
21
11
1
11
1
11
AA A A A
AA A A A
M A AA A A A
AA A A A
A
A
A A A A
A AA A A AA A A
A A A
A A A AA A
A A31 14
3411
41 1341 12 41 1442 43 44
11 11 11
0
A AA
A
A AA A A AA A A
A A A
MA/CS 375 Fall 2003 16
M1 A Has Zeros Below The Diagonal
11 12 13 14
21 22 23 2413
31 32 33 34
41 42 43 444
11 12 13 14
21 1321 12 21 1422 23 24
11 11 11
31 12 31 1332 33
11 11
1 0 0 0
1 0 0
0 1 0
0 0 1
0
0
21
11
1
11
1
11
AA A A A
AA A A A
M A AA A A A
AA A A A
A
A
A A A A
A AA A A AA A A
A A A
A A A AA A
A A31 14
3411
41 1341 12 41 1442 43 44
11 11 11
0
A AA
A
A AA A A AA A A
A A A
MA/CS 375 Fall 2003 17
LU Factorization cont.
A suitable candidate for M2 is the matrix ( 4 by 4 case):
1
2321
22
1
241
22
1 0 0 0
0 1 0 0
0 1 0
0 0 1
M AM M A
M A
M A
MA/CS 375 Fall 2003 18
LU Factorization cont.
A suitable candidate for M3 is the matrix ( 4 by 4 case):
3
1
431
33
1 0 0 0
0 1 0 0
0 0 1 0
0 0 1
2
2
MM M A
M M A
MA/CS 375 Fall 2003 19
LU Factorization cont.• So in this case: U=(M3M2M1)A is upper triangle
• Each of the M matrices is lower triangle
• The inverse of M2 looks like: • Using the fact that the inverse of the each M matrix just requires us to negate the below diagonal terms
• A = (M3M2M1)-1 U = (M1)-1 (M2)-1 (M3)-1 U • Define L = (M1)-1 (M2)-1 (M3)-1 and we are done since the product of lower matrices is a lower matrix
1
32121
22
1
421
22
1 0 0 0
0 1 0 0
0 1 0
0 0 1
M AM M A
M A
M A
MA/CS 375 Fall 2003 20
Matlab Routine ForLU Factorization
MA/CS 375 Fall 2003 21
Matlab Routine ForLU Factorization
• Build L and U
• A is modified in place
• Total cost O(N3) ( we can get this by noticing the triply nested loop)
MA/CS 375 Fall 2003 22
Matlab Routine ForLU Factorization
And Solution of Ax=b
• Build rhs
• Solve Ly=b
• Solve Ux=y
MA/CS 375 Fall 2003 23
Team Exercise1) Code up the factorization
2) Test your code to make sure it gives thecorrect answer as Matlab • Make sure abs(LU – Aorig) is small• Make sure abs(x - Aorig\b) is small
3) Make sure the routine actually finished without the break.
4) Time the code for an N=400 case
5) Compare with the timings for [Lmat,Umat] = lu(Aorig);
ASK IF YOU DO NOT UNDERSTAND ANY PART OF THIS!!
MA/CS 375 Fall 2003 24
When Basic LU Does Not Work
• Remember that we are required to compute 1/A(i,i) for all the successive diagonal terms
• What happens when we get to the i’th diagonal term and this turns out to be 0 or very small ??
• At any point we can swap the i’th row with one lower down which does not have a zero (or small entry) in the i’th column.
•This is known as partial pivoting.
MA/CS 375 Fall 2003 25
Partial PivotingRemember before that we constructed a setof M matrices so that is upper triangle
Now we construct a sequence of M matricesand P matrices so that:
is upper triangle.
(MN-1…. M4 M3 M2 M1)A
(MN-1 PN-1 …. M4 P4 M3 P3 M2 P2 M1 P1)A
MA/CS 375 Fall 2003 26
LU with Partial Pivoting
MA/CS 375 Fall 2003 27
LU with Partial Pivoting
1) find pivot
2) swap rows
3) Do elimination
MA/CS 375 Fall 2003 28
LU with Partial Pivoting
1) build matrix A2) call our LU routine3) check that:
A(piv,:) = L*U
4) Looks good
MA/CS 375 Fall 2003 29
Team Exercise• Make sure you all have a copy of the .m files on your
laptops (courtesy of Coutsias)• Locate GEpiv• For N=100,200,400,600
– build a random NxN matrix A– time how long GEpiv takes to factorize A
• end• Plot the cputime taken per test against N using loglog• On the same graph plot N^3 using loglog• Observation