Numerical Computation
Lecture 8: Matrix Algorithm Analysis
United International College
Review Presentation
• 10 minutes for review presentation.
Review
• During our Last Class we covered:– Gauss-Jordan Method for finding Inverses
Today
• We will cover:– Operation count for Gaussian Elimination, LU
Factorization– Accuracy of Matrix Methods– Readings: • Pav, section 3.4.1• Moler, section 2.8
Operation Count for Gaussian Elimination
• How many floating point operations (+-*/) are used by the Gaussian Elimination algorithm?
• Definition: Flop = floating point operation. We will consider a division to be equivalent to a multiplication, and a subtraction equivalent to an addition.
• Thus, 2/3 = 2*(1/3) will be considered a multiplication.
• 2-3 = 2 + (-3) will be considered an addition.
Operation Count for Gaussian Elimination
• In Gaussian Elimination we use row operations to reduce
• to
Operation Count for Gaussian Elimination
• Consider the number of flops needed to zero out the entries below the first pivot a11 .
Operation Count for Gaussian Elimination
• First a multiplier is computed for each row below the first row. This requires (n-1) multiplies.
m = A(i,k)/A(k,k); • Then in each row below row 1 the algorithm
performs n multiplies and n adds. (A(i,j) = A(i,j) - m*A(k,j);) • Thus, there is a total of (n-1) + (n-1)*2*n flops for
this step of Gaussian Elimination. • For k=1 algorithm uses 2n2 –n -1 flops
Operation Count for Gaussian Elimination
• For k =2, we zero out the column below a22 .• There are (n-2) rows below this pivot, so this
takes 2(n-1)2 –(n-1) -1 flops. • For k -3, we would have 2(n-2)2 –(n-2) -1 flops,
and so on. • To complete Gaussian Elimination, it will take
In flops, where
Operation Count for Gaussian Elimination
• Now,
• So, In = (2/6)n(n+1)(2n+1) – (1/2)n(n+1) – n
= [(1/3)(2n+1)-(1/2)]*n(n+1) – n = [(2/3)n – (1/6)] * n(n+1) - n = (2/3)n3 + (lower power terms in n)• Thus, the number of flops for Gaussian
Elimination is O(n3).
Operation Count for LU Factorization
• In the algorithm for LU Factorization, we only do the calculations described above to compute L and U. This is because we save the multipliers (m) and store them to create L.
• So, the number of flops to create L and U is O(n3).
Operation Count for using LU to solve Ax = b
• Once we have factored A into LU, we do the following to solve Ax = b:
• Solve the two equations:• Lz = b• Ux = z
• How many flops are needed to do this?
Operation Count for using LU to solve Ax = b
• To solve Lz=b we use forward substitution
z =
z1 = b1 , so we use 0 flops to find z1.z2 = b2 – l21 *z1 , so we use 2 flops to find z2 .z3 = b3 – l31 *z1 – l32 *z2 , so we use 4 flops to find z2 , and so on.
Operation Count for using LU to solve Ax = b
• To solve Lz=b we use forward substitution
z =
• Totally, 0+2+4+ … + 2*(n-1)= 2*(1+2+…+(n-1)) = 2*(1/2)*(n-1)(n) = n2 – n. • So, the number of flops for forward substitution is
O(n2).
Operation Count for using LU to solve Ax = b
• To solve Ux=z we use backward substitution
• A similar analysis to that of forward substitution shows that the number of flops for backward substitution is also O(n2).
• Thus, the number of flops for using LU to solve Ax=b is O(n2).
Summary of Two Methods
• Gaussian Elimination requires O(n3) flops to solve the linear system Ax = b.
• To factor A = LU requires O(n3) flops • Once we have factored A = LU, then, using L
and U to solve Ax = b requires O(n2) flops.
• Suppose we have to solve Ax = b for a given matrix A, but for many different b vectors. What is the most efficient way to do this?
Accuracy of Matrix Methods
• Our algorithms are used to find the solution x to the system Ax=b.
• But, how close to the exact solution is the computed solution?
• Let x* be the computed solution and x be the exact solution.
Accuracy of Matrix Methods• Definition: The error e is defined to be
e = x - x*
• Definition: The residual r is defined to be
r = b – Ax*
• Note: These two quantities may be quite different!
Accuracy of Matrix Methods
• Consider the system:
• Using Gaussian Elimination with partial pivoting, we swap rows 1 and 2
217.0254.0
563.0780.0659.0913.0
2
1
xx
254.0217.0
659.0913.0563.0780.0
2
1
xx
Accuracy of Matrix Methods
• Suppose we had a computer with just 3 digit accuracy. The first multiplier would be:
• Subtracting 0.854*row 1 from row 2, we get
Accuracy of Matrix Methods
• Solving this using back substitution gives
• So,
Accuracy of Matrix Methods
• The exact solution is
• Thus, the error is
• The error is bigger than the solution!!
000.2557.0
000.1000.1)443.0(000.1
*xxe
Accuracy of Matrix Methods
• The residual is
• The residual is very small, but the error is very large!
Accuracy of Matrix Methods
• Theorem (sort of): Gaussian elimination with partial pivoting is guaranteed to produce small residuals.
• Why is the error in our example so large?
Accuracy of Matrix Methods
• If we did Gaussian Elimination with much higher accuracy (more than 3 digits) we would see that the row reduction produces:
• This matrix is very close to being singular (why?)• The relationship between the size of the residual and
the size of the error is determined in part by a quantity known as the condition number of the matrix, which is the subject of our next lecture.