8/13/2019 Chapter 8 - Solving Linear Systems of Equations
1/55
Solving Linear Systems of Equations
Lecture8
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
2/55
Whats the big deal? ....cost
Look at 1D:
3 equations
3 unknowns
each unknown coupled to its neighbor
2x1 x2 = 5.8x1 2x2 x3 = 13.9
x2 2x3 = 0.03
or
2 1 0
1 2 10 1 2
x1
x2x3
5.8
13.90.03
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
3/55
Easy in 1d
2 1
. . .1 2 1
. . .
1 2
npoints in the grid3 (n 2) + 2+ 2 or about 3nnonzeros in the matrix
trididagonal (easy)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
4/55
2D: harder
8 1 1
. . .1 1 8 1 1
. . .
1 1 8
npoints in one direction
n2 points in grid
about 9 nonzeros in each row
about 9n2 nonzeros in the matrix
n-banded (harder...we will see)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
5/55
3D: hardest
npoints in one direction
n3 points in grid
about 27 nonzeros in each row
about 27n3 nonzeros in thematrix
n2
-banded (yikes!)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
6/55
Applications get harder and harder...
courtesy of LLNL courtesy of TrueGrid
courtesy of Rice courtesy of Warwick U.
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
7/55
Solving is a problem...
dim unknowns storage example
1D n 3n 10 100 10002D n2 9n2 102 104 106
3D n2 27n3 103 106 109
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
8/55
Moore...
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
9/55
Whats the problem?
humans: milliFLOPS
hand calculators: 10 FLOPS
desktops: a few GFLOPS (109 FLOPS)
look at the basic operations!
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
10/55
vec-vec, mat-vec, mat-mat
inner product ofu and v both[n 1]
=uT
v=u1v1+ +unvn
nmultiplies,n 1 additions
O(n)flops
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
11/55
vec-vec, mat-vec, mat-mat
mat-vec ofA ([n n]) andu ([n 1])
1 for i= 1, . . . , n2 fo r j= 1, . . . , n
3 v(i) =a(i,j)u(j) +v(i)4 en d
5 end
n2 multiplies,n2 additions
O(n2)flops
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
12/55
vec-vec, mat-vec, mat-mat
mat-mat ofA ([n n]) andB ([n n])
1 for j= 1, . . . , n2 fo r i= 1, . . . , n3 for k= 1, . . . , n4 C(k,j) =A(k, i)B(i,j) +C(k,j)5 end
6 en d
7 end
n3 multiplies,n3 additions
O(n3)flops
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
13/55
vec-vec, mat-vec, mat-mat
Operation FLOPS
uTv O(n)Au O(n2)AB O(n3)
matlab test
four tests:
matrix-matrix multiply
inner product
matrix-vector multiply
nmatrix-vector multiplies
order them...fastest to slowest
testflop.m
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
14/55
Gaussian Elimination
Solving Diagonal Systems
Solving Triangular Systems
Gaussian Elimination Without Pivoting Hand Calculations Cartoon Version The Algorithm
Gaussian Elimination with Pivoting Row or Column Interchanges, or Both Implementation
Solving Systems with the Backslash Operator
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
15/55
Solving Diagonal Systems
The system defined by
A=
1 0 0
0 3 0
0 0 5
b=
1
6
15
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
16/55
Solving Diagonal Systems
The system defined by
A=
1 0 0
0 3 0
0 0 5
b=
1
6
15
is equivalent tox1 = 1
3x2 = 65x3 = 15
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
17/55
Solving Diagonal Systems
The system defined by
A=
1 0 0
0 3 0
0 0 5
b=
1
6
15
is equivalent tox1 = 1
3x2 = 65x3 = 15
The solution is
x1 = 1 x2 = 6
3 = 2 x3 =
15
5 = 3
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
18/55
Solving Diagonal Systems (1)
Listing 1: Diagonal System Solution
1 given A, b2 for i= 1 . . . n3 xi =bi/ai,i
4 end
InM:
1 >> A = ... % A is a diagona l m at ri x
2 >> b = ...
3 >> x = b./ diag (A)
This is theonlyplace where element-by-element division (.*) has anything todo with solving linear systems of equations.
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
19/55
Operations?
Try...
Sketch out an operation count to solvea diagonal system of equations...
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
20/55
Operations?
Try...
Sketch out an operation count to solvea diagonal system of equations...
cheap!
one divisionn times O(n)FLOPS
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
21/55
Triangular Systems (1)
The generic lower and upper triangular matrices are
L=
l11 0 0l21 l22 0...
. . . ...
ln1 lnn
and
U=
u11 u12 u1n0 u22 u2n...
. . . ...
0 unn
The triangular systemsLy=b Ux=c
are easily solved byforward substitutionand backward substitution,respectively
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
22/55
Solving Triangular Systems (2)
A=
2 1 2
0 3 2
0 0 4
b= 9
1
8
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
23/55
Solving Triangular Systems (3)
A=
2 1 2
0 3 2
0 0 4
b= 9
1
8
is equivalent to
2x1 + x2 + 2x3 = 93x2 + 2x3 = 1
4x3 = 8
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
24/55
Solving Triangular Systems (4)
A=
2 1 2
0 3 2
0 0 4
b= 9
1
8
is equivalent to
2x1 + x2 + 2x3 = 93x2 + 2x3 = 1
4x3 = 8
Solve in backward order (last equation is solved first)
x3 = 84
= 2 x2 = 13
(1+ 2x3)= 33
= 1
x1 = 1
2(9x22x3)=
4
2 = 2
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
25/55
Solving Triangular Systems (5)
Solving forx1, x2, . . . , xn for a lower triangular system is called forwardsubstitution.
1 given L, b2 x1 =b1/113 for i= 2 . . . n
4 s= bi5 for j= 1 . . . i 16 s= si,jxj7 end
8 xi =s/i,i9 end
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
26/55
Solving Triangular Systems (6)
Solving forx1, x2, . . . , xn for a lower triangular system is called forwardsubstitution.
1 given L, b2 x1 =b1/113 for i= 2 . . . n
4 s= bi5 for j= 1 . . . i 16 s= si,jxj7 end
8 xi =s/i,i9 end
Using forward or backward substitution is sometimes referred to as performingatriangular solve.
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
27/55
Operations?
Try...Sketch out an operation count to solvea triangular system of equations...
O ?
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
28/55
Operations?
Try...Sketch out an operation count to solvea triangular system of equations...
cheap!
begin in the bottom corner: 1 div
row -2: 1 mult, 1 add, 1 div, or 3 FLOPS
row -3: 2 mult, 2 add, 1 div, or 5 FLOPS
row -4: 3 mult, 3 add, 1 div, or 7 FLOPS
..
.row -j: about 2jFLOPS
Total FLOPS? n
j=12j= 2n(n+1)
2 orO(n2)FLOPS
G i Eli i i
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
29/55
Gaussian Elimination
Goal is to transform an arbitrary, square system into the equivalent uppertriangular system so that it may be easily solved with backward substitution.Theformal solutionto Ax =b, whereA is ann nmatrix is
x=
A
1
bInM:
1 >> A = ...
2 >> b = ...
3 >> x = A\b
G i Eli i ti H d C l l ti (1)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
30/55
Gaussian Elimination Hand Calculations (1)
Solve
x1+3x2 = 5
2x1+4x2 = 6
Subtract 2 times the first equation from the second equation
x1+3x2 = 5
2x2 = 4
This equation is now in triangular form, and can be solved by backwardsubstitution.
G i Eli i ti H d C l l ti (2)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
31/55
Gaussian Elimination Hand Calculations (2)
The elimination phase transforms the matrix and right hand side to anequivalent system
x1+3x2 = 5
2x1+4x2 = 6
x1+3x2 = 5
2x2 = 4
The two systems have the same solution. The right hand system is uppertriangular.Solve the second equation forx2
x2 = 4
2 = 2
Substitute the newly found value ofx2 into the first equation and solve forx1.
x1 = 5 (3)(2) = 1
Ga ssian Elimination Hand Calc lations (3)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
32/55
Gaussian Elimination Hand Calculations (3)
When performing Gaussian Elimination by hand, we can avoid copying the xiby using a shorthand notation.For example, to solve:
A=
3 2 1
6 6 7
3 4 4
b=
1
7
6
Form theaugmentedsystem
A=[A b]=
3 2 1
6 6 7
3 4 4
1
7
6
The vertical bar inside the augmented matrix is just a reminder that the lastcolumn is theb vector.
Gaussian Elimination Hand Calculations (4)
http://goforward/http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
33/55
Gaussian Elimination Hand Calculations (4)
Add 2 times row 1 to row 2, and add (1 times) row 1 to row 3
A(1) =
3 2 1
0 2 5
0 2 3
1
9
7
Subtract (1 times) row 2 from row 3
A(2) =
3 2 1
0 2 5
0 0 2
1
9
2
Gaussian Elimination Hand Calculations (5)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
34/55
Gaussian Elimination Hand Calculations (5)
The transformed system is now in upper triangular form
A(2) = 3 2 10 2 5
0 0 2
19
2
Solve by back substitution to get
x3 = 22
= 1
x2 = 1
2(9 5x3)= 2
x1 = 1
3(1 2x2+x3)= 2
Gaussian Elimination Cartoon Version (1)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
35/55
Gaussian Elimination Cartoon Version (1)
Start with the augmented system
x x x x xx x x x xx x x x xx x x x x
Thexs represent numbers, they are not necessarily the same values.Begin elimination using the first row as the pivot rowand the first element ofthe first row as the pivot element
x x x x x
x x x x xx x x x xx x x x x
Gaussian Elimination Cartoon Version (2)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
36/55
Gaussian Elimination Cartoon Version (2)Eliminate elements under the pivot element in the first column. x indicates avalue that has been changed once.
x x x x xx x x x x
x x x x x
x x x x x
x x x x x0 x x x x
x x x x x
x x x x x
x x x x x
0 x x x x
0 x x x x
x x x x x
x x x x x
0 x x x x
0 x x x x
0 x x x x
Gaussian Elimination Cartoon Version (3)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
37/55
Gaussian Elimination Cartoon Version (3)
The pivot element is now the diagonal element in the second row. Eliminate
elements under the pivot element in the second column. x
indicates a valuethat has been changed twice.
x x x x x
0 x x x x
0 x x x x
0 x x x x
x x x x x
0 x x x x
0 0 x x x
0 x x x x
x x x x x
0 x x x x
0 0 x x x
0 0 x x x
Gaussian Elimination Cartoon Version (4)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
38/55
Gaussian Elimination Cartoon Version (4)
The pivot element is now the diagonal element in the third row. Eliminateelements under the pivot element in the third column. x indicates a valuethat has been changed three times.
x x x x x
0 x x x x
0 0 x x x
0 0 x x x
x x x x x
0 x x x x
0 0 x x x
0 0 0 x x
Gaussian Elimination Cartoon Version (5)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
39/55
Gaussian Elimination Cartoon Version (5)
Summary
Gaussian Elimination is an orderly process of transforming an augmentedmatrix into an equivalent upper triangular form.
The elimination operation is
akj = akj (aki/aii)aij
Elimination requires three nested loops.
The result of the elimination phase is represented by the image below.
x x x x x
x x x x xx x x x x
x x x x x
x x x x x
0 x
x
x
x
0 0 x x x
0 0 0 x x
Gaussian Elimination Algorithm
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
40/55
Gaussian Elimination Algorithm
Listing 2: Gaussian Elimination
1 form A= [Ab]2
3 for i= 1 . . . n 14 for k= i+ 1 . . . n5 for j= i . . . n+ 1
6 akj = akj (aki/aii)aij7 end
8 end
9 end
GEshowTheGEshowfunction in the NMM toolbox uses Gaussianeliminationtosolve asystem of equations. GEshowis intended fordemonstration purposes only.
The Need for Pivoting (1)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
41/55
The Need for Pivoting (1)
Solve:
A=
2 4 2 2
1 2 4 3
3 3 8 2
1 1 6 3
b=
4
5
7
7
Note that there is nothing wrong with this system.A
is full rank. The solutionexists and is unique.Form the augmented system.
2 4 2 2
1 2 4 3
3 3 8 21 1 6 3
4
5
77
The Need for Pivoting (2)
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
42/55
The Need for Pivoting (2)
Subtract 1/2times the first row from the second row,add 3/2times the first row to the third row,add 1/2times the first row to the fourth row.The result of these operations is:
2 4 2 20 0 5 2
0 3 5 5
0 3 5 4
47
1
5
Thenextstage of Gaussian elimination will not work because there is a zeroin thepivotlocation, a22.
The Need for Pivoting (3)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
43/55
The Need for Pivoting (3)
Swap second and fourth rows of the augmented matrix.
2 4 2 2
0 3 5 4
0 3 5 5
0 0 5 2
4
5
1
7
Continue with elimination: subtract (1 times) row 2 from row 3.
2 4 2 2
0 3 5 4
0 0 0
1
0 0 5 2
4
5
4
7
The Need for Pivoting (4)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
44/55
g ( )
Another zero has appear in the pivot position. Swap row 3 and row 4.
2 4 2 2
0 3 5 4
0 0 5 2
0 0 0 1
4
5
7
4
The augmented system is now ready for backward substitution.
Pivoting Strategies
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
45/55
g g
Partial Pivoting: Exchange only rows
Exchanging rows does not affect the order of the xi
For increased numerical stability, make sure the largest possible pivotelement is used. This requires searching in the partial column below thepivot element.
Partial pivoting is usually sufficient.
Partial Pivoting
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
46/55
g
To avoid division by zero, swap the row having the zero pivot with one of the
rows below it.
0
*
Rows completed in
forward elimination.
Rows to search for a
more favorable pivot
element.
Row with zero pivot element
To minimize the effect of roundoff, always choose the row that puts the largestpivot element on the diagonal, i.e., findip such that|aip,i|= max(|ak,i|)fork=i, . . . , n
Pivoting Strategies (2)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
47/55
g g ( )
Full (or Complete) Pivoting:Exchangebothrows and columns
Column exchange requires changing the order of thexi
For increased numerical stability, make sure the largest possible pivotelement is used. This requires searching in the pivot row, andin all rowsbelow the pivot row, starting the pivot column.
Full pivoting is less susceptible to roundoff, but the increase in stabilitycomes at a cost of more complex programming (not a problem if you usea library routine) and an increase in work associated with searching anddata movement.
Full Pivoting
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
48/55
0
*
Rows completed inforward elimination.
Columns to search for a morefavorable pivot element.
Row with zero pivot element
Rows to search for a
more favorable pivotelement.
*
Gauss Elimination with Partial Pivoting
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
49/55
Listing 3: Gaussian Elimination with Partial Pivoting
1 form A= [Ab]2 for i= 1 . . . n 13 find ip s uc h t ha t4 max(|aipi|) max(|aki|) for k= i . . . n
5 exchange row ip w it h r ow i6 for k= i+ 1 . . . n7 for j= i . . . n+ 18 ak,j = ak,j (ak,i/ai,i)ai,j9 end
0 end
1 end
Gauss Elimination with Partial Pivoting
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
50/55
GEshow
TheGEshowfunction in the NMM toolbox uses naiveGaussianeliminationwithout pivoting to solve a system of equations.
GEpivshowTheGEpivshowfunction in the NMM toolbox uses Gaussian elimination withpartial pivoting to solve a system of equations. GEpivshowis intendedfordemonstration purposes only.
GEshowand GEpivshoware for demonstration purposes only. They are also aconvenient way to check your hand calculations.
The Backslash Operator (1)
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
51/55
Consider the scalar equation
5x= 20 = x= (5)120
The extension to a system of equations is, of course
Ax= b = x=A1b
whereA1bis the formal solution toAx = bIn Mnotation the system is solved with
1
x = A\b
The Backslash Operator (2)
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
52/55
Given ann nmatrixA, and ann 1vectorb the \operator performs asequence of tests on theA matrix. Mattempts to solve the system with
the method that gives the least roundoff and the fewest operations.WhenA is ann nmatrix:
1 MexaminesA to see if it is a permutation of a triangular system
If so, the appropriate triangular solve is used.
2 MexaminesA to see if itappearsto be symmetric and positivedefinite.
If so, Mattempts a Cholesky factorization
and two triangular solves.
3 If the Cholesky factorization fails, or ifA does not appear to besymmetric,
Mattempts anLUfactorizationand two triangular solves.
Limits on Numerical Solution toAx=b
http://find/http://goback/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
53/55
Machine Limitations
RAM requirements grow as O(n2)
flop count grows asO(n3
)The time for data movement is an important speed bottleneck on modernsystems
An architecture observation...
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
54/55
Disk
FPU CPU
internal bus
cache
system bus
RAM
network
Computer
Data Access Time:
shortest
longest
Stand alone computer
Computer Computer
http://find/8/13/2019 Chapter 8 - Solving Linear Systems of Equations
55/55
Operation FLOPSuTv O(n)Au O(n2)AB O(n3)
quiz...
order each in terms ofspeed
order each in terms ofefficiency
http://find/