L. Vandenberghe ECE133A (Fall 2021)
7. Linear equations
โข QR factorization method
โข factor and solve
โข LU factorization
7.1
QR factorization and matrix inverse
QR factorization of nonsingular matrix
every nonsingular ๐ด โ R๐ร๐ has a QR factorization
๐ด = ๐๐
โข ๐ โ R๐ร๐ is orthogonal (๐๐๐ = ๐๐๐ = ๐ผ)
โข ๐ โ R๐ร๐ is upper triangular with positive diagonal elements
Inverse from QR factorization: the inverse ๐ดโ1 can be written as
๐ดโ1 = (๐๐ )โ1 = ๐ โ1๐โ1 = ๐ โ1๐๐
Linear equations 7.2
Solving linear equations by QR factorization
Algorithm: to solve ๐ด๐ฅ = ๐ with nonsingular ๐ด โ R๐ร๐,
1. factor ๐ด as ๐ด = ๐๐
2. compute ๐ฆ = ๐๐๐
3. solve ๐ ๐ฅ = ๐ฆ by back substitution
Complexity: 2๐3 + 3๐2 โ 2๐3 flops
โข QR factorization: 2๐3
โข matrix-vector multiplication: 2๐2
โข back substitution: ๐2
Linear equations 7.3
Multiple right-hand sides
consider ๐ sets of linear equations with the same coefficient matrix ๐ด:
๐ด๐ฅ1 = ๐1, ๐ด๐ฅ2 = ๐2, . . . , ๐ด๐ฅ๐ = ๐๐
โข equivalently, solve ๐ด๐ = ๐ต where ๐ต is ๐ ร ๐ matrix with columns ๐1, . . . , ๐๐
โข can be solved in 2๐3 + 3๐๐2 flops if we reuse the factorization ๐ด = ๐๐
โข for ๐ ๏ฟฝ ๐, cost is roughly equal to cost of solving one equation (2๐3)
Application: to compute ๐ดโ1, solve the matrix equation ๐ด๐ = ๐ผ
โข equivalent to ๐ equations
๐ ๐ฅ1 = ๐๐๐1, ๐ ๐ฅ2 = ๐๐๐2, . . . , ๐ ๐ฅ๐ = ๐๐๐๐
(๐ฅ๐ is column ๐ of ๐ and ๐๐๐๐ is transpose of ๐th row of ๐)
โข complexity is 2๐3 + ๐3 = 3๐3 (here the 2nd term ๐3 is not negligible)
Linear equations 7.4
Factorโsolve approach
to solve ๐ด๐ฅ = ๐, first write ๐ด as a product of โsimpleโ matrices
๐ด = ๐ด1๐ด2 ยท ยท ยท ๐ด๐
then solve (๐ด1๐ด2 ยท ยท ยท ๐ด๐)๐ฅ = ๐ by solving ๐ equations
๐ด1๐ง1 = ๐, ๐ด2๐ง2 = ๐ง1, . . . , ๐ด๐โ1๐ง๐โ1 = ๐ง๐โ2, ๐ด๐๐ฅ = ๐ง๐โ1
Examples
โข QR factorization: ๐ = 2, ๐ด = ๐๐
โข LU factorization (this lecture)
โข Cholesky factorization (later)
Linear equations 7.5
Complexity of factorโsolve method
#flops = ๐ + ๐
โข ๐ is complexity of factoring ๐ด as ๐ด = ๐ด1๐ด2 ยท ยท ยท ๐ด๐ (factorization step)
โข ๐ is complexity of solving the ๐ equations for ๐ง1, ๐ง2, . . . ๐ง๐โ1, ๐ฅ (solve step)
โข usually ๐ ๏ฟฝ ๐
Example: solving linear equations using the QR factorization
๐ = 2๐3, ๐ = 3๐2
Linear equations 7.6
LU factorization
LU factorization without pivoting
๐ด = ๐ฟ๐
โข ๐ฟ unit lower triangular, ๐ upper triangular
โข does not always exist (even if ๐ด is nonsingular)
LU factorization (with row pivoting)
๐ด = ๐๐ฟ๐
โข ๐ permutation matrix, ๐ฟ unit lower triangular, ๐ upper triangular
โข exists if and only if ๐ด is nonsingular (see later)
Complexity: (2/3)๐3 if ๐ด is ๐ ร ๐
Linear equations 7.7
Solving linear equations by LU factorization
Algorithm: to solve ๐ด๐ฅ = ๐ with nonsingular ๐ด of size ๐ ร ๐
1. factor ๐ด as ๐ด = ๐๐ฟ๐ ((2/3)๐3 flops)
2. solve (๐๐ฟ๐)๐ฅ = ๐ in three steps
(a) permutation: ๐ง1 = ๐๐๐ (0 flops)(b) forward substitution: solve ๐ฟ๐ง2 = ๐ง1 (๐2 flops)(c) back substitution: solve ๐๐ฅ = ๐ง2 (๐2 flops)
Complexity: (2/3)๐3 + 2๐2 โ (2/3)๐3 flops
this is the standard method for solving ๐ด๐ฅ = ๐
Linear equations 7.8
Multiple right-hand sides
two equations with the same matrix ๐ด (nonsingular and ๐ ร ๐):
๐ด๐ฅ = ๐, ๐ด๐ฅ = ๏ฟฝฬ๏ฟฝ
โข factor ๐ด once
โข forward/back substitution to get ๐ฅ
โข forward/back substitution to get ๐ฅ
complexity: (2/3)๐3 + 4๐2 โ (2/3)๐3
Exercise: propose an efficient method for solving
๐ด๐ฅ = ๐, ๐ด๐๐ฅ = ๏ฟฝฬ๏ฟฝ
Linear equations 7.9
LU factorization and matrix inverse
suppose ๐ด is nonsingular and ๐ ร ๐, with LU factorization
๐ด = ๐๐ฟ๐
โข inverse from LU factorization
๐ดโ1 = (๐๐ฟ๐)โ1 = ๐โ1๐ฟโ1๐๐
โข gives interpretation of solve step: we evaluate
๐ฅ = ๐ดโ1๐ = ๐โ1๐ฟโ1๐๐๐
in three steps๐ง1 = ๐๐๐, ๐ง2 = ๐ฟโ1๐ง1, ๐ฅ = ๐โ1๐ง2
Linear equations 7.10
Computing the inverse
solve ๐ด๐ = ๐ผ column by column
โข one LU factorization of ๐ด: 2๐3/3 flops
โข ๐ solve steps: 2๐3 flops
โข total: (8/3)๐3 flops
slightly faster methods exist that exploit structure in right-hand side ๐ผ
Conclusion: do not solve ๐ด๐ฅ = ๐ by multiplying ๐ดโ1 with ๐
Linear equations 7.11
LU factorization without pivoting
[๐ด11 ๐ด1,2:๐๐ด2:๐,1 ๐ด2:๐,2:๐
]=
[1 0
๐ฟ2:๐,1 ๐ฟ2:๐,2:๐
] [๐11 ๐1,2:๐0 ๐2:๐,2:๐
]=
[๐11 ๐1,2:๐
๐11๐ฟ2:๐,1 ๐ฟ2:๐,1๐1,2:๐ + ๐ฟ2:๐,2:๐๐2:๐,2:๐
]Recursive algorithm
โข determine first row of ๐ and first column of ๐ฟ
๐11 = ๐ด11, ๐1,2:๐ = ๐ด1,2:๐, ๐ฟ2:๐,1 =1๐ด11
๐ด2:๐,1
โข factor the (๐ โ 1) ร (๐ โ 1)-matrix ๐ด2:๐,2:๐ โ ๐ฟ2:๐,1๐1,2:๐ as
๐ด2:๐,2:๐ โ ๐ฟ2:๐,1๐1,2:๐ = ๐ฟ2:๐,2:๐๐2:๐,2:๐
this is an LU factorization (without pivoting) of size (๐ โ 1) ร (๐ โ 1)
Linear equations 7.12
Example
LU factorization (without pivoting) of
๐ด =
8 2 94 9 46 7 9
write as ๐ด = ๐ฟ๐ with ๐ฟ unit lower triangular, ๐ upper triangular
๐ด =
8 2 94 9 46 7 9
=
1 0 0๐ฟ21 1 0๐ฟ31 ๐ฟ32 1
๐11 ๐12 ๐130 ๐22 ๐230 0 ๐33
Linear equations 7.13
Example
โข first row of ๐, first column of ๐ฟ:8 2 94 9 46 7 9
=
1 0 01/2 1 03/4 ๐ฟ32 1
8 2 90 ๐22 ๐230 0 ๐33
โข second row of ๐, second column of ๐ฟ:[
9 47 9
]โ[
1/23/4
] [2 9
]=
[1 0๐ฟ32 1
] [๐22 ๐230 ๐33
][
8 โ1/211/2 9/4
]=
[1 0
11/16 1
] [8 โ1/20 ๐33
]โข third row of ๐: ๐33 = 9/4 + 11/32 = 83/32
Conclusion
๐ด =
8 2 94 9 46 7 9
=
1 0 01/2 1 03/4 11/16 1
8 2 90 8 โ1/20 0 83/32
Linear equations 7.14
Not every nonsingular ๐ด can be factored as ๐ด = ๐ฟ๐
๐ด =
1 0 00 0 20 1 โ1
=
1 0 0๐ฟ21 1 0๐ฟ31 ๐ฟ32 1
๐11 ๐12 ๐130 ๐22 ๐230 0 ๐33
โข first row of ๐, first column of ๐ฟ:
1 0 00 0 20 1 โ1
=
1 0 00 1 00 ๐ฟ32 1
1 0 00 ๐22 ๐230 0 ๐33
โข second row of ๐, second column of ๐ฟ:[
0 21 โ1
]=
[1 0๐ฟ32 1
] [๐22 ๐230 ๐33
]๐22 = 0, ๐23 = 2, ๐ฟ32 ยท 0 = 1 ?
Linear equations 7.15
LU factorization (with row pivoting)
if ๐ด is ๐ ร ๐ and nonsingular, then it can be factored as
๐ด = ๐๐ฟ๐
๐ is a permutation matrix, ๐ฟ is unit lower triangular, ๐ is upper triangular
โข not unique; there may be several possible choices for ๐, ๐ฟ, ๐
โข interpretation: permute the rows of ๐ด and factor ๐๐๐ด as ๐๐๐ด = ๐ฟ๐
โข also known as Gaussian elimination with partial pivoting (GEPP)
โข complexity: (2/3)๐3 flops
we skip the details of calculating ๐, ๐ฟ, ๐
Linear equations 7.16
Example
0 5 52 9 06 8 8
=
0 0 10 1 01 0 0
1 0 01/3 1 00 15/19 1
6 8 80 19/3 โ8/30 0 135/19
the factorization is not unique; the same matrix can be factored as
0 5 52 9 06 8 8
=
0 1 01 0 00 0 1
1 0 00 1 03 โ19/5 1
2 9 00 5 50 0 27
Linear equations 7.17
Effect of rounding error
[10โ5 1
1 1
] [๐ฅ1๐ฅ2
]=
[10
]
solution:๐ฅ1 =
โ11 โ 10โ5 , ๐ฅ2 =
11 โ 10โ5
โข let us solve using LU factorization for the two possible permutations:
๐ =
[1 00 1
]or ๐ =
[0 11 0
]โข we round intermediate results to four significant decimal digits
Linear equations 7.18
First choice: ๐ = ๐ผ (no pivoting)[10โ5 1
1 1
]=
[1 0
105 1
] [10โ5 1
0 1 โ 105
]โข ๐ฟ, ๐ rounded to 4 significant decimal digits
๐ฟ =
[1 0
105 1
], ๐ =
[10โ5 1
0 โ105
]โข forward substitution[
1 0105 1
] [๐ง1๐ง2
]=
[10
]=โ ๐ง1 = 1, ๐ง2 = โ105
โข back substitution[10โ5 1
0 โ105
] [๐ฅ1๐ฅ2
]=
[1
โ105
]=โ ๐ฅ1 = 0, ๐ฅ2 = 1
error in ๐ฅ1 is 100%Linear equations 7.19
Second choice: interchange rows[1 1
10โ5 1
]=
[1 0
10โ5 1
] [1 10 1 โ 10โ5
]โข ๐ฟ, ๐ rounded to 4 significant decimal digits
๐ฟ =
[1 0
10โ5 1
], ๐ =
[1 10 1
]โข forward substitution[
1 010โ5 1
] [๐ง1๐ง2
]=
[01
]=โ ๐ง1 = 0, ๐ง2 = 1
โข backward substitution[1 10 1
] [๐ฅ1๐ฅ2
]=
[01
]=โ ๐ฅ1 = โ1, ๐ฅ2 = 1
error in ๐ฅ1, ๐ฅ2 is about 10โ5
Linear equations 7.20
Conclusion: rounding error and LU factorization
โข for some choices of ๐, small errors in the algorithm can cause very large errorsin the solution
โข this is called numerical instability: for the first choice of ๐ in the example, thealgorithm is unstable; for the second choice of ๐, it is stable
โข from numerical analysis: there is a simple rule for selecting a good permutation
(we skip the details, since we skipped the details of the factorization)
Linear equations 7.21
Sparse linear equations
if ๐ด is sparse, it is usually factored as
๐ด = ๐1๐ฟ๐๐2
๐1 and ๐2 are permutation matrices
โข interpretation: permute rows and columns of ๐ด and factor ๏ฟฝฬ๏ฟฝ = ๐๐1 ๐ด๐๐2
๏ฟฝฬ๏ฟฝ = ๐ฟ๐
โข choice of ๐1 and ๐2 greatly affects the sparsity of ๐ฟ and ๐: several heuristicmethods exist for selecting good permutations
โข in practice: #flops ๏ฟฝ (2/3)๐3; exact value depends on ๐, number of nonzeroelements, sparsity pattern
Linear equations 7.22
Conclusion
different levels of detail in understanding how linear equation solvers work
Highest level
โข x = A \ b costs (2/3)๐3
โข more efficient than x = inv(A) * b
Intermediate level: factorization step ๐ด = ๐๐ฟ๐ followed by solve step
Lowest level: details of factorization ๐ด = ๐๐ฟ๐
โข for most applications, level 1 is sufficient
โข in some situations (e.g., multiple right-hand sides) level 2 is useful
โข level 3 is important for experts who write numerical libraries
Linear equations 7.23