Embed Size (px)
Citation preview
1
Systems of Linear Equations
Gauss-Jordan Elimination and
LU Decomposition
2
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
3
Gauss-Jordan Elimination
)3(3
)3(2
)3(1
)2(33
)2(3
)1(22
)2(2
11)2(
1
)2(3
)2(33
)2(2
)1(22
)2(111
)2(3
)2(33
)1(2
)1(23
)1(22
)1(1
)1(1311
)1(3
)1(33
)1(32
)1(2
)1(23
)1(22
1131211
3333231
2232221
1131211
1
1
1
1
1
1
c
c
c
bb
ab
ab
ba
ba
ba
ba
baa
baa
baa
baa
baaa
baaa
baaa
baaa
Similar to the Gauss elimination except
1. Elimination is applied to all equations (excluding the pivot equation) instead of just the subsequent equations.
2. All rows are normalized by dividing them by their pivot elements.
3. No back substitution is required.
4
Pitfalls and Improvements
• Pitfalls: Same as those found in the Gauss elimination.– Division-by-zero, round-off error, and ill-
conditioned systems.
• Improvement strategies: Same as those used in the Gauss elimination – Use pivoting and scaling to avoid division-by-
zero and to reduce round-off error.
5
Computing CostWhich of Gauss elimination and Gauss-Jordan elimination involves more FLOPS?
Gauss Elimination Gauss-Jordan Elimination
Elimination Step
Forward Elimination – only needs to eliminate the coefficients below the diagonal.
Cost ~ 2n3/3
Needs to eliminate coefficients below and above the diagonal.
Cost ~ 2 * (2n3/3)
Substitution Step
Back Substitution
Cost ~ O(n2)
No substitution step
Total 2n3/3 + O(n2) 4n3/3
(More costly when n is big)
6
Question
What is
0
0
11A
333231
232221
1312111
333231
232221
131211
ˆˆˆ
ˆˆˆ
ˆˆˆ
aaa
aaa
aaa
aaa
aaa
aaa
AA
0
0
1
ˆ
ˆ
ˆ
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
0
0
1
31
21
11
31
21
11
31
21
111
31
21
111
a
a
a
a
a
a
a
a
a
a
a
a
AAIAAAA
31
21
11
3
2
1
1
3
2
1
ˆ
ˆ
ˆ
ofcolumnfirstthegivessystemthisSolving
0
0
1
a
a
a
x
x
x
A
x
x
x
A
?
7
Computing Matrix Inverse
• Gauss-Jordan
333231
232221
131211
333231
232221
131211
ˆˆˆ1
ˆˆˆ1
ˆˆˆ1
1
1
1
aaa
aaa
aaa
aaa
aaa
aaa
333231
232221
1312111
333231
232221
131211
ˆˆˆ
ˆˆˆ
ˆˆˆ
aaa
aaa
aaa
aaa
aaa
aaa
AA
• Gauss Elimination
1
0
0
ˆ
ˆ
ˆ
0
1
0
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
33
23
13
32
22
12
31
21
11
a
a
a
a
a
a
a
a
a
AAA
Solve
Let
8
Summary• Algorithms
– Gauss elimination• Forward elimination + back substitution
– Gauss-Jordan elimination• Calculate the computing cost of Gauss Elimination in
terms of FLOPS
• To avoid division-by-zero, use pivoting (partial or complete).
• To reduce round-off error, use pivoting and scaling.
• If a system is ill-conditioned, then substituting solution back to the system to check if the solution is accurate is not reliable.– How to determine if a matrix is ill-conditioned?
9
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
10
3x1 + 2x2 + x3 = 6
2x2 + 3x3 = 7
2x3 = 4
Step 1: Solve for x3
2x3 = 4 => x3 = 4/2 = 2
Step 2: Solve for x2
2x2 + 3(2) = 7
=> x2 = [7 – 3(2)] / 2 = 0.5
Step 3: Solve for x1
3x1 + 2(0.5) + (2) = 6
=> x1 = [6 – 2(0.5) – 2] / 3 = 1
Use to solve the system Ux = b where U is an upper triangular matrix.
33
2322
131211
00
0
u
uu
uuu
U
Back SubstitutionExample:
11
2x1 = 4
3x1 + 2x2 = 7
x1 + 2x2 + 3x3 = 6
Step 1: Solve for x1
2x1 = 4 => x1 = 4/2 = 2
Step 2: Solve for x2
3(2) + 2x2 = 7
=> x2 = [7 – 3(2)] / 2 = 0.5
Step 3: Solve for x3
(2) + 2(0.5) + 3x3 = 6
x3 = [6 – 2 – 2(0.5)] / 3 = 1
Use to solve the system Lx = b where L is a lower triangular matrix.
Forward SubstitutionExample:
33231
2221
11
0
00
lll
ll
l
L
12
LU DecompositionIdea: To solve Ax = b• Step 1: Decompose A into L and U such that
A = LU where
33
2322
131211
33231
2221
11
00
00
00
u
uu
uuu
lll
ll
l
UL
We will discuss later how to compute L and U when given A. For now, let's assume we can!
• Step 2: Solve LUx = b using forward and back substitution.
13
Solving LUx = b
)onsubstitutiback
usingCompute(:2
)onsubstitutiforward
usingCompute(:1
)(Let
xyUx
ybLy
Uxyby
UxL
Step
Step
14
Decomposing A using Gauss Elimination
U is formed by applying the forward elimination of Gauss Elimination (without including the right hand side vector)
"33
'23
'22
131211
a
aa
aaa
U
22
3232
11
3131
11
2121
3231
21 '
',,where
1
01
001
a
af
a
af
a
af
ff
f
L
L is formed by the factors used to eliminate each elements during the forward elimination process as
15
Decomposing A into L and U -- Example
102.03.0
3.071.0
2.01.03
A
102.03.0
293333.000333.70
2.01.03
1row 2row ,3
1.02121 ff
02.1019.00
293333.000333.70
2.01.03
1row 3row ,3
3.03131 ff
1st iteration (1st column), eliminating a21
1st iteration (1st column), eliminating a22
16
Decomposing A into L and U -- Example
99996.92.03.0
3.070999999.0
2.01.03
LU
0120.1000
293333.000333.70
2.01.03
2row 3row ,00333.7
19.03232 ff
2st iteration (2st column), eliminating a32
0120.1000
293333.000333.70
2.01.03
10271300.010000000.0
0103333333.0
001
UL
Thus we can express A as LU where
Due to round-off error
17
LU Decomposition• In LU decomposition, the l and u values are
obtained from
11
1
1
1
1
ijulaul
ijulau
j
kkjikijjjij
i
kkjikijij
Note: These are just mathematical expressions representing the steps involved in Gauss Elimination.
18
Compact Representation
For U, the lower part are all 0's.
"333231
'23
'2221
131211
aff
aaf
aaa
1
01
001
3231
21
ff
fL
"33
'23
'22
131211
a
aa
aaa
U
For L, the diagonal elements are all 1's and the upper part are all 0's.
When implementing the algorithm, we can store both U and L back into A.
19
Forward and Back Substitution
121for1 ,,nnia
xayx
a
yx
ii
n
ij jiji
inn
nn
To solve Ly = b, we use forward substitution
To solve Ux = y, we use back substitution
"333231
'23
'2221
131211
aff
aaf
aaaNote: Here the aij represents the coefficients of the resulting matrix A produced by the LU Decomposition algorithm.
nibayybyi
jjijii ...,,3,2for and
1
111
20
Effect of Pivoting
333231
2322
13
0
00
aaa
aa
a
A
If we directly apply Gauss elimination algorithm (with pivoting), we will need to swap row 3 with row 1 and eventually yield L and U as
11
2322
333231
00
0
100
010
001
a
aa
aaa
UL
Let
Pivot row in 1st iteration
21
Effect of Pivoting
If we are given a vector b, can we solve Ax = b as LUx = b?
11
2322
333231
333231
2322
13
00
0
100
010
001
0
00
a
aa
aaa
aaa
aa
a
ULA
No! We can't because we have swapped the rows during the elimination step.
In implementing the LU decomposition, we have to remember how the rows were swapped so that we can reorder the elements in x and b accordingly.
22
333231
232221
131211
aaa
aaa
aaa
333231
232221
131211
aaa
aaa
aaa
Addressing the issue of row swapping in implementation
i o[i]
1 1
2 2
3 3
To remember how the rows were swapped, we can introduce an array, o[] , to store the indexes to the rows.Initially, o[i] = i for i = 1, 2, …, N.
Whenever we need to swap rows, we only swap the indexes stored in o[]. For example, after swapping row 1 with row 3
i o[i]
1 3
2 2
3 1
23
Addressing the issue of row swapping in implementation
If rows were swapped, A ≠A'. However, o[i] tells us that row i in A corresponds row o[i] in A'. To solve Ax = b, we can first reorder the elements in x and b to produce x' and b' as
x'i = xo[i] and b'i = bo[i]
and then solve A'x' = b' as L'U'x' = b'.
Note: In actual implementation, we do not need to explicitly create x' and b'. We can refer to their elements directly as xo[i] and bo[i].
AL'
U' A'
LU Decomposition with pivoting
=
24
Illustration: Using o[]
i o[i]
1 3
2 2
3 1
25.0
5.0
75.05.1
5.03
124
31
21
31
21
f
f
f
f
221
142
124
124
142
221
1 2 2
2 4 1
4 2 1
Array A
i o[i]
1 1
2 2
3 3
1 2 2
2 4 1
4 2 1
Array A
After swapping row 3 with row 1
Initial states.
Pick row 3 as pivot row
i o[i]
1 3
2 2
3 1
0.25 1.5 0.75
0.5 3 0.5
4 2 1
Array A
f31 is calculated as A[o[3],1] / A[o[1],1] and the result is stored back to A[o[3],1].
After the 1st iteration.
Implementation
25
Pseudocode for LU Decomposition
// Assume arrays start with index 1 instead of 0.// a: Coef. of matrix A; 2-D array. Upon successful// completion, it contains the coefficients of // both L and U.// b: Coef. of vector b; 1-D array// n: Dimension of the system of equations// x: Coef. of vector x (to store the solution)// tol: Tolerance; smallest possible scaled // pivot allowed.// er: Pass back -1 if matrix is singular. // (Reference var.) LUDecomp(a, b, n, x, tol, er) { Declare s[n] // An n-element array for storing scaling factors Declare o[n] // Use as indexes to pivot rows. // oi or o(i) stores row number of the ith pivot row. er = 0 Decompose(a, n, tol, o, s, er) if (er != -1) Substitute(a, o, n, b, x)}
26
Pseudocode for decomposing A into L and UDecompose(a, n, tol, o, s, er) {
for i = 1 to n { // Find scaling factors
o[i] = i
s[i] = abs(a[i,1])
for j = 2 to n
if (abs(a[i,j]) > s[i])
s[i] = abs(a[i,j])
}
for k = 1 to n-1 { Pivot(a, o, s, n, k) // Locate the kth pivot row
// Check for singular or near-singular cases if (abs(a[o[k],k]) / s[o[k]]) < tol) { er = -1 return } // Continue next page
27
for i = k+1 to n { factor = a[o[i],k] / a[o[k],k]
// Instead of storing the factors // in another matrix (2D array) L, // We reuse the space in A to store // the coefficients of L. a[o[i],k] = factor // Eliminate the coefficients at column j // in the subsequent rows for j = k+1 to n a[o[i],j] = a[o[i],j] – factor *
a[o[k],j] }
} // end of "for k" loop from previous page
// Check for singular or near-singular cases if (abs(a[o[n],n]) / s[o[n]]) < tol) er = -1}
28
Psuedocode for finding the pivot rowPivot(a, o, s, n, k) { // Find the largest scaled coefficient in column k p = k // p is the index to the pivot row big = abs(a[o[k],k]) / s[o[k]]) for i = k+1 to n { dummy = abs(a[o[i],k] / s[o(i)]) if (dummy > big) { big = dummy p = i } }
// Swap row k with the pivot row by swapping the // indexes. The actual rows remain unchanged dummy = o[p] o[p] = o[k] o[k] = dummy}
29
Psuedocode for solving LUx = bSubstitute(a, o, n, b, x) { Declare y[n] y[o[1]] = b[o[1]] for i = 2 to n { sum = b[o[i]] for j = 1 to i-1 sum = sum – a[o[i],j] * b[o[j]] y[o[i]] = sum }
x[n] = y[o[n]] / a[o[n],n] for i = n-1 downto 1 { sum = 0 for j = i+1 to n sum = sum + a[o[i],j] * x[j] x[i] = (y[o[i]] – sum) / a[o[i],i] }}
Forwardsubstitution
Backsubstitution
30
About the LU Decomposition Pseudocode
• If the LU decomposition algorithm is to be used to solve Ax = b for various b's, then both array "a" and array "o" have to be saved.
• Subroutine substitute() can be applied repeatedly for various b's.
• In subroutine substitute(), we can also reuse the space in "b" to store the elements of "y" instead of declaring the new array "y".
31
Question
What is the cost to decompose A into L and U?
32
LU Decomposition vs. Gauss Elimination
To solve Ax = bi, i = 1, 2, 3, …, K
LU Decomposition
Compute L and U once – O(n3)
Forward and back substitution – O(n2)
Total cost = O(n3) + K * O(n2)
Gauss Elimination
Total cost: K * O(n3)
33
Computing Matrix Inverse
333231
232221
1312111
333231
232221
131211
ˆˆˆ
ˆˆˆ
ˆˆˆ
aaa
aaa
aaa
aaa
aaa
aaa
AA
1
0
0
ˆ
ˆ
ˆ
0
1
0
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
33
23
13
32
22
12
31
21
11
a
a
a
a
a
a
a
a
a
LULULU
Solve
Let
First, decompose A into L and U, then
34
Pseudocode for finding matrix inverse using LU Decomposition// a: Given matrix A; 2-D array.// ai: Stores the coefficients of A-1
// n: Dimension of A// tol: Smallest possible scaled pivot allowed.// er: Pass back -1 if matrix is singular.LU_MatrixInverse(a, ai, n, tol, er) { Declare s[n], o[n], b[n], x[n]
Decompose(a, n, tol, o, s, er) if (er != -1) { for i = 1 to n { for j = 1 to n // Construct the unit vector b[j] = 0 b[i] = 1 Substitute(a, o, n, b, x)
for j = 1 to n ai[j,i] = x[j] } }}
35
Two types of LU decomposition• Doolittle Decomposition or factorization
nn
n
n
nnnnnn
n
n
u
uu
uuu
ll
l
aaa
aaa
aaa
00
0
1
01
001
222
11211
21
21
21
22221
11211
A
• Crout Decomposition
100
10
1
0
00
2
112
21
2221
11
21
22221
11211
n
n
nnnnnnnn
n
n
u
uu
lll
ll
l
aaa
aaa
aaa
A
36
Crout Decomposition
1
1
1
1
1
1
1
11
11
11
1for
for
1...,,3,2for
...,,2for
...,,2,1for
n
kknnknnnn
j
iikjijkjjjk
j
kkjikijij
jj
ii
ulal
jkulalu
jiulal
nj
njl
au
nial
37
Summary
• LU Decomposition Algorithm
• LU Decomposition vs. Gauss Elimination– Similarity– Advantages (Disadvantages?)– Computing Cost
