MA/CS 375 Fall 2003 1

MA/CS 375

Fall 2003

Lecture 19

MA/CS 375 Fall 2003 2

Matrices and Systems

MA/CS 375 Fall 2003 3

Upper Triangular Systems

11 1 12 2 1 1

1 22 2 2 2

1 2

...

0 ...

0 0 ...

N N

N N

NN N N

x x x b

x x x b

x x x b

U U U

U U

U

MA/CS 375 Fall 2003 4

Upper Triangular Systems

Volunteer: tell us how to solve this? (hint: this is not difficult)

11 1 12 2 1 1

1 22 2 2 2

1 2

...

0 ...

0 0 ...

N N

N N

NN N N

x x x b

x x x b

x x x b

U U U

U U

U

MA/CS 375 Fall 2003 5

Example: Upper Triangular Systems

11 1 12 2 13 3 1

1 22 2 23 3 2

1 2 33 3 3

0

0 0

x x x b

x x x b

x x x b

U U U

U U

U

MA/CS 375 Fall 2003 6

Upper Triangular Systems1) Solve the Nth equation

2) Use the value from step 1 to solve the (N-1)th equation

3) Use the values from steps 1 and 2 to solve the (N-2)th equation

4) keep going until you solve the whole system.

MA/CS 375 Fall 2003 7

Backwards Substitution

1 1 1,1, 1

2 2 2, 1 1 2,2, 2

1

1 1 1,21,1

1

1

1

1

1

N NNN

N N N N NN N

N N N N N N N NN N

j N

i i ij jj iii

j N

j jj

x b

x b x

x b x x

x b x

x b x

U

UU

U UU

UU

UU

MA/CS 375 Fall 2003 8

Matlab Code for Backward

Substitution

1

1 j N

i i ij jj iii

x b x

U

U

MA/CS 375 Fall 2003 9

Testing Backwards Substitution

MA/CS 375 Fall 2003 10

Finite Precision Effects

• Notice that x(1) depends on the values of x(2),x(3),…,x(N)

• Remembering that round off rears its ugly head when we deal with (small+large) numbers

• Volunteer to design a U matrix and b vector, which do not give good answers when used in this algorithm.

• (goal is to give you some intuition for cases which will break an algorithm)

MA/CS 375 Fall 2003 11

Team Exercise

• Who can build a “reasonable”, non-singular, U matrix and b vector which has the worst answer for U\b

• 10 Minutes only

MA/CS 375 Fall 2003 12

LU Factorization• By now you should be persuaded that it is not difficult to solve a problem of the form Lx=b or Ux=b using forward/backward elimination

• Suppose we are given a matrix A and we are able to find a lower triangular matrix L and an upper triangular matrix U such that:

A = LU

• Then to solve Ax=b we can solve two simple problems:

• Ly = b• Ux = y • (sanity check LUx = L(Ux) = Ly =b)

MA/CS 375 Fall 2003 13

Example LU Factorization

• Suppose we are given:

• Then we can write A = LU where:

• Let’s check that:

2 3

1 2

A

1 0

0.5 1

L

2 3

0 0.5

U

1 0 2 3 1*2 0*0 1*3 0*0.5 2 3

0.5 1 0 0.5 0.5*2 1*0 0.5*3 1*0.5 1 2

LU

MA/CS 375 Fall 2003 14

LU Factorization (in words)• Construct a sequence of multiplier matrices (M1, M2, M3, M4,…., MN-1) such that:

(MN-1…. M4 M3 M2 M1)A is upper triangle.

•A suitable candidate for M1 is the matrix ( 4 by 4 case):

13

4

1 0 0 0

1 0 0

0 1 0

0 0 1

21

11

1

11

1

11

A

A

M A

A

A

A

MA/CS 375 Fall 2003 15

Check to See What M1 Does to A

11 12 13 14

21 22 23 2413

31 32 33 34

41 42 43 444

11 12 13 14

21 1321 12 21 1422 23 24

11 11 11

31 12 31 1332 33

11 11

1 0 0 0

1 0 0

0 1 0

0 0 1

0

0

21

11

1

11

1

11

AA A A A

AA A A A

M A AA A A A

AA A A A

A

A

A A A A

A AA A A AA A A

A A A

A A A AA A

A A31 14

3411

41 1341 12 41 1442 43 44

11 11 11

0

A AA

A

A AA A A AA A A

A A A

MA/CS 375 Fall 2003 16

M1 A Has Zeros Below The Diagonal

11 12 13 14

21 22 23 2413

31 32 33 34

41 42 43 444

11 12 13 14

21 1321 12 21 1422 23 24

11 11 11

31 12 31 1332 33

11 11

1 0 0 0

1 0 0

0 1 0

0 0 1

0

0

21

11

1

11

1

11

AA A A A

AA A A A

M A AA A A A

AA A A A

A

A

A A A A

A AA A A AA A A

A A A

A A A AA A

A A31 14

3411

41 1341 12 41 1442 43 44

11 11 11

0

A AA

A

A AA A A AA A A

A A A

MA/CS 375 Fall 2003 17

LU Factorization cont.

A suitable candidate for M2 is the matrix ( 4 by 4 case):

1

2321

22

1

241

22

1 0 0 0

0 1 0 0

0 1 0

0 0 1

M AM M A

M A

M A

MA/CS 375 Fall 2003 18

LU Factorization cont.

A suitable candidate for M3 is the matrix ( 4 by 4 case):

3

1

431

33

1 0 0 0

0 1 0 0

0 0 1 0

0 0 1

2

2

MM M A

M M A

MA/CS 375 Fall 2003 19

LU Factorization cont.• So in this case: U=(M3M2M1)A is upper triangle

• Each of the M matrices is lower triangle

• The inverse of M2 looks like: • Using the fact that the inverse of the each M matrix just requires us to negate the below diagonal terms

• A = (M3M2M1)-1 U = (M1)-1 (M2)-1 (M3)-1 U • Define L = (M1)-1 (M2)-1 (M3)-1 and we are done since the product of lower matrices is a lower matrix

1

32121

22

1

421

22

1 0 0 0

0 1 0 0

0 1 0

0 0 1

M AM M A

M A

M A

MA/CS 375 Fall 2003 20

Matlab Routine ForLU Factorization

MA/CS 375 Fall 2003 21

Matlab Routine ForLU Factorization

• Build L and U

• A is modified in place

• Total cost O(N3) ( we can get this by noticing the triply nested loop)

MA/CS 375 Fall 2003 22

Matlab Routine ForLU Factorization

And Solution of Ax=b

• Build rhs

• Solve Ly=b

• Solve Ux=y

MA/CS 375 Fall 2003 23

Team Exercise1) Code up the factorization

2) Test your code to make sure it gives thecorrect answer as Matlab • Make sure abs(LU – Aorig) is small• Make sure abs(x - Aorig\b) is small

3) Make sure the routine actually finished without the break.

4) Time the code for an N=400 case

5) Compare with the timings for [Lmat,Umat] = lu(Aorig);

ASK IF YOU DO NOT UNDERSTAND ANY PART OF THIS!!

MA/CS 375 Fall 2003 24

When Basic LU Does Not Work

• Remember that we are required to compute 1/A(i,i) for all the successive diagonal terms

• What happens when we get to the i’th diagonal term and this turns out to be 0 or very small ??

• At any point we can swap the i’th row with one lower down which does not have a zero (or small entry) in the i’th column.

•This is known as partial pivoting.

MA/CS 375 Fall 2003 25

Partial PivotingRemember before that we constructed a setof M matrices so that is upper triangle

Now we construct a sequence of M matricesand P matrices so that:

is upper triangle.

(MN-1…. M4 M3 M2 M1)A

(MN-1 PN-1 …. M4 P4 M3 P3 M2 P2 M1 P1)A

MA/CS 375 Fall 2003 26

LU with Partial Pivoting

MA/CS 375 Fall 2003 27

LU with Partial Pivoting

1) find pivot

2) swap rows

3) Do elimination

MA/CS 375 Fall 2003 28

LU with Partial Pivoting

1) build matrix A2) call our LU routine3) check that:

A(piv,:) = L*U

4) Looks good

MA/CS 375 Fall 2003 29

Team Exercise• Make sure you all have a copy of the .m files on your

laptops (courtesy of Coutsias)• Locate GEpiv• For N=100,200,400,600

– build a random NxN matrix A– time how long GEpiv takes to factorize A

• end• Plot the cputime taken per test against N using loglog• On the same graph plot N^3 using loglog• Observation