Chapter 8 - Solving Linear Systems of Equations

8/13/2019 Chapter 8 - Solving Linear Systems of Equations

1/55

Solving Linear Systems of Equations

Lecture8
http://find/


2/55

Whats the big deal? ....cost

Look at 1D:

3 equations

3 unknowns

each unknown coupled to its neighbor

2x1 x2 = 5.8x1 2x2 x3 = 13.9

x2 2x3 = 0.03

or

2 1 0

1 2 10 1 2

x1

x2x3

5.8

13.90.03
http://find/


3/55

Easy in 1d

2 1

. . .1 2 1

. . .

1 2

npoints in the grid3 (n 2) + 2+ 2 or about 3nnonzeros in the matrix

trididagonal (easy)
http://find/


4/55

2D: harder

8 1 1

. . .1 1 8 1 1

. . .

1 1 8

npoints in one direction

n2 points in grid

about 9 nonzeros in each row

about 9n2 nonzeros in the matrix

n-banded (harder...we will see)
http://find/


5/55

3D: hardest

npoints in one direction

n3 points in grid

about 27 nonzeros in each row

about 27n3 nonzeros in thematrix

n2

-banded (yikes!)
http://find/


6/55

Applications get harder and harder...

courtesy of LLNL courtesy of TrueGrid

courtesy of Rice courtesy of Warwick U.
http://find/


7/55

Solving is a problem...

dim unknowns storage example

1D n 3n 10 100 10002D n2 9n2 102 104 106

3D n2 27n3 103 106 109
http://find/


8/55

Moore...
http://find/


9/55

Whats the problem?

humans: milliFLOPS

hand calculators: 10 FLOPS

desktops: a few GFLOPS (109 FLOPS)

look at the basic operations!
http://find/


10/55

vec-vec, mat-vec, mat-mat

inner product ofu and v both[n 1]

=uT

v=u1v1+ +unvn

nmultiplies,n 1 additions

O(n)flops
http://find/


11/55


mat-vec ofA ([n n]) andu ([n 1])

1 for i= 1, . . . , n2 fo r j= 1, . . . , n

3 v(i) =a(i,j)u(j) +v(i)4 en d

5 end

n2 multiplies,n2 additions

O(n2)flops
http://find/


12/55


mat-mat ofA ([n n]) andB ([n n])

1 for j= 1, . . . , n2 fo r i= 1, . . . , n3 for k= 1, . . . , n4 C(k,j) =A(k, i)B(i,j) +C(k,j)5 end

6 en d

7 end

n3 multiplies,n3 additions

O(n3)flops
http://find/


13/55


Operation FLOPS

uTv O(n)Au O(n2)AB O(n3)

matlab test

four tests:

matrix-matrix multiply

inner product

matrix-vector multiply

nmatrix-vector multiplies

order them...fastest to slowest

testflop.m
http://find/


14/55

Gaussian Elimination

Solving Diagonal Systems

Solving Triangular Systems

Gaussian Elimination Without Pivoting Hand Calculations Cartoon Version The Algorithm

Gaussian Elimination with Pivoting Row or Column Interchanges, or Both Implementation

Solving Systems with the Backslash Operator
http://find/http://goback/


15/55


The system defined by

A=

1 0 0

0 3 0

0 0 5

b=

1

6

15
http://find/


16/55



A=

1 0 0

0 3 0

0 0 5

b=

1

6

15

is equivalent tox1 = 1

3x2 = 65x3 = 15
http://find/


17/55



A=

1 0 0

0 3 0

0 0 5

b=

1

6

15

is equivalent tox1 = 1

3x2 = 65x3 = 15

The solution is

x1 = 1 x2 = 6

3 = 2 x3 =

15

5 = 3
http://find/


18/55

Solving Diagonal Systems (1)

Listing 1: Diagonal System Solution

1 given A, b2 for i= 1 . . . n3 xi =bi/ai,i

4 end

InM:

1 >> A = ... % A is a diagona l m at ri x

2 >> b = ...

3 >> x = b./ diag (A)

This is theonlyplace where element-by-element division (.*) has anything todo with solving linear systems of equations.
http://find/


19/55

Operations?

Try...

Sketch out an operation count to solvea diagonal system of equations...
http://find/


20/55

Operations?

Try...

Sketch out an operation count to solvea diagonal system of equations...

cheap!

one divisionn times O(n)FLOPS


21/55

Triangular Systems (1)

The generic lower and upper triangular matrices are

L=

l11 0 0l21 l22 0...

. . . ...

ln1 lnn

and

U=

u11 u12 u1n0 u22 u2n...

. . . ...

0 unn

The triangular systemsLy=b Ux=c

are easily solved byforward substitutionand backward substitution,respectively


22/55

Solving Triangular Systems (2)

A=

2 1 2

0 3 2

0 0 4

b= 9

1

8
http://find/


23/55


A=

2 1 2

0 3 2

0 0 4

b= 9

1

8

is equivalent to

2x1 + x2 + 2x3 = 93x2 + 2x3 = 1

4x3 = 8
http://find/


24/55


A=

2 1 2

0 3 2

0 0 4

b= 9

1

8

is equivalent to

2x1 + x2 + 2x3 = 93x2 + 2x3 = 1

4x3 = 8

Solve in backward order (last equation is solved first)

x3 = 84

= 2 x2 = 13

(1+ 2x3)= 33

= 1

x1 = 1

2(9x22x3)=

4

2 = 2
http://find/


25/55


Solving forx1, x2, . . . , xn for a lower triangular system is called forwardsubstitution.

1 given L, b2 x1 =b1/113 for i= 2 . . . n

4 s= bi5 for j= 1 . . . i 16 s= si,jxj7 end

8 xi =s/i,i9 end


26/55


Solving forx1, x2, . . . , xn for a lower triangular system is called forwardsubstitution.

1 given L, b2 x1 =b1/113 for i= 2 . . . n

4 s= bi5 for j= 1 . . . i 16 s= si,jxj7 end

8 xi =s/i,i9 end

Using forward or backward substitution is sometimes referred to as performingatriangular solve.
http://find/


27/55

Operations?

Try...Sketch out an operation count to solvea triangular system of equations...

O ?


28/55

Operations?

Try...Sketch out an operation count to solvea triangular system of equations...

cheap!

begin in the bottom corner: 1 div

row -2: 1 mult, 1 add, 1 div, or 3 FLOPS



..

.row -j: about 2jFLOPS

Total FLOPS? n

j=12j= 2n(n+1)

2 orO(n2)FLOPS

G i Eli i i
http://find/


29/55

Gaussian Elimination

Goal is to transform an arbitrary, square system into the equivalent uppertriangular system so that it may be easily solved with backward substitution.Theformal solutionto Ax =b, whereA is ann nmatrix is

x=

A

1

bInM:

1 >> A = ...

2 >> b = ...

3 >> x = A\b

G i Eli i ti H d C l l ti (1)
http://find/


30/55

Gaussian Elimination Hand Calculations (1)

Solve

x1+3x2 = 5

2x1+4x2 = 6

Subtract 2 times the first equation from the second equation

x1+3x2 = 5

2x2 = 4

This equation is now in triangular form, and can be solved by backwardsubstitution.

G i Eli i ti H d C l l ti (2)
http://find/


31/55


The elimination phase transforms the matrix and right hand side to anequivalent system

x1+3x2 = 5

2x1+4x2 = 6

x1+3x2 = 5

2x2 = 4

The two systems have the same solution. The right hand system is uppertriangular.Solve the second equation forx2

x2 = 4

2 = 2

Substitute the newly found value ofx2 into the first equation and solve forx1.

x1 = 5 (3)(2) = 1

Ga ssian Elimination Hand Calc lations (3)
http://find/


32/55


When performing Gaussian Elimination by hand, we can avoid copying the xiby using a shorthand notation.For example, to solve:

A=

3 2 1

6 6 7

3 4 4

b=

1

7

6

Form theaugmentedsystem

A=[A b]=

3 2 1

6 6 7

3 4 4

1

7

6

The vertical bar inside the augmented matrix is just a reminder that the lastcolumn is theb vector.

http://goforward/http://find/


33/55


Add 2 times row 1 to row 2, and add (1 times) row 1 to row 3

A(1) =

3 2 1

0 2 5

0 2 3

1

9

7

Subtract (1 times) row 2 from row 3

A(2) =

3 2 1

0 2 5

0 0 2

1

9

2

http://find/


34/55


The transformed system is now in upper triangular form

A(2) = 3 2 10 2 5

0 0 2

19

2

Solve by back substitution to get

x3 = 22

= 1

x2 = 1

2(9 5x3)= 2

x1 = 1

3(1 2x2+x3)= 2

Gaussian Elimination Cartoon Version (1)
http://find/


35/55


Start with the augmented system

x x x x xx x x x xx x x x xx x x x x

Thexs represent numbers, they are not necessarily the same values.Begin elimination using the first row as the pivot rowand the first element ofthe first row as the pivot element

x x x x x

x x x x xx x x x xx x x x x

http://find/


36/55

Gaussian Elimination Cartoon Version (2)Eliminate elements under the pivot element in the first column. x indicates avalue that has been changed once.

x x x x xx x x x x

x x x x x

x x x x x

x x x x x0 x x x x

x x x x x

x x x x x

x x x x x

0 x x x x

0 x x x x

x x x x x

x x x x x

0 x x x x

0 x x x x

0 x x x x

http://find/


37/55


The pivot element is now the diagonal element in the second row. Eliminate

elements under the pivot element in the second column. x

indicates a valuethat has been changed twice.

x x x x x

0 x x x x

0 x x x x

0 x x x x

x x x x x

0 x x x x

0 0 x x x

0 x x x x

x x x x x

0 x x x x

0 0 x x x

0 0 x x x

http://find/


38/55


The pivot element is now the diagonal element in the third row. Eliminateelements under the pivot element in the third column. x indicates a valuethat has been changed three times.

x x x x x

0 x x x x

0 0 x x x

0 0 x x x

x x x x x

0 x x x x

0 0 x x x

0 0 0 x x

http://find/


39/55


Summary

Gaussian Elimination is an orderly process of transforming an augmentedmatrix into an equivalent upper triangular form.

The elimination operation is

akj = akj (aki/aii)aij

Elimination requires three nested loops.

The result of the elimination phase is represented by the image below.

x x x x x

x x x x xx x x x x

x x x x x

x x x x x

0 x

x

x

x

0 0 x x x

0 0 0 x x

Gaussian Elimination Algorithm


40/55

Gaussian Elimination Algorithm

Listing 2: Gaussian Elimination

1 form A= [Ab]2

3 for i= 1 . . . n 14 for k= i+ 1 . . . n5 for j= i . . . n+ 1

6 akj = akj (aki/aii)aij7 end

8 end

9 end

GEshowTheGEshowfunction in the NMM toolbox uses Gaussianeliminationtosolve asystem of equations. GEshowis intended fordemonstration purposes only.

The Need for Pivoting (1)
http://find/


41/55


Solve:

A=

2 4 2 2

1 2 4 3

3 3 8 2

1 1 6 3

b=

4

5

7

7

Note that there is nothing wrong with this system.A

is full rank. The solutionexists and is unique.Form the augmented system.

2 4 2 2

1 2 4 3

3 3 8 21 1 6 3

4

5

77



42/55


Subtract 1/2times the first row from the second row,add 3/2times the first row to the third row,add 1/2times the first row to the fourth row.The result of these operations is:

2 4 2 20 0 5 2

0 3 5 5

0 3 5 4

47

1

5

Thenextstage of Gaussian elimination will not work because there is a zeroin thepivotlocation, a22.

http://find/


43/55


Swap second and fourth rows of the augmented matrix.

2 4 2 2

0 3 5 4

0 3 5 5

0 0 5 2

4

5

1

7

Continue with elimination: subtract (1 times) row 2 from row 3.

2 4 2 2

0 3 5 4

0 0 0

1

0 0 5 2

4

5

4

7

http://find/


44/55

g ( )

Another zero has appear in the pivot position. Swap row 3 and row 4.

2 4 2 2

0 3 5 4

0 0 5 2

0 0 0 1

4

5

7

4

The augmented system is now ready for backward substitution.

Pivoting Strategies
http://find/


45/55

g g

Partial Pivoting: Exchange only rows

Exchanging rows does not affect the order of the xi

For increased numerical stability, make sure the largest possible pivotelement is used. This requires searching in the partial column below thepivot element.

Partial pivoting is usually sufficient.

Partial Pivoting
http://find/


46/55

g

To avoid division by zero, swap the row having the zero pivot with one of the

rows below it.

0

*

Rows completed in

forward elimination.

Rows to search for a

more favorable pivot

element.

Row with zero pivot element

To minimize the effect of roundoff, always choose the row that puts the largestpivot element on the diagonal, i.e., findip such that|aip,i|= max(|ak,i|)fork=i, . . . , n

Pivoting Strategies (2)
http://find/


47/55

g g ( )

Full (or Complete) Pivoting:Exchangebothrows and columns

Column exchange requires changing the order of thexi

For increased numerical stability, make sure the largest possible pivotelement is used. This requires searching in the pivot row, andin all rowsbelow the pivot row, starting the pivot column.

Full pivoting is less susceptible to roundoff, but the increase in stabilitycomes at a cost of more complex programming (not a problem if you usea library routine) and an increase in work associated with searching anddata movement.

Full Pivoting
http://find/


48/55

0

*

Rows completed inforward elimination.

Columns to search for a morefavorable pivot element.

Row with zero pivot element

Rows to search for a

more favorable pivotelement.

*

Gauss Elimination with Partial Pivoting
http://find/


49/55

Listing 3: Gaussian Elimination with Partial Pivoting

1 form A= [Ab]2 for i= 1 . . . n 13 find ip s uc h t ha t4 max(|aipi|) max(|aki|) for k= i . . . n

5 exchange row ip w it h r ow i6 for k= i+ 1 . . . n7 for j= i . . . n+ 18 ak,j = ak,j (ak,i/ai,i)ai,j9 end

0 end

1 end

Gauss Elimination with Partial Pivoting


50/55

GEshow

TheGEshowfunction in the NMM toolbox uses naiveGaussianeliminationwithout pivoting to solve a system of equations.

GEpivshowTheGEpivshowfunction in the NMM toolbox uses Gaussian elimination withpartial pivoting to solve a system of equations. GEpivshowis intendedfordemonstration purposes only.

GEshowand GEpivshoware for demonstration purposes only. They are also aconvenient way to check your hand calculations.

The Backslash Operator (1)
http://find/


51/55

Consider the scalar equation

5x= 20 = x= (5)120

The extension to a system of equations is, of course

Ax= b = x=A1b

whereA1bis the formal solution toAx = bIn Mnotation the system is solved with

1

x = A\b

The Backslash Operator (2)


52/55

Given ann nmatrixA, and ann 1vectorb the \operator performs asequence of tests on theA matrix. Mattempts to solve the system with

the method that gives the least roundoff and the fewest operations.WhenA is ann nmatrix:

1 MexaminesA to see if it is a permutation of a triangular system

If so, the appropriate triangular solve is used.

2 MexaminesA to see if itappearsto be symmetric and positivedefinite.

If so, Mattempts a Cholesky factorization

and two triangular solves.

3 If the Cholesky factorization fails, or ifA does not appear to besymmetric,

Mattempts anLUfactorizationand two triangular solves.

Limits on Numerical Solution toAx=b


53/55

Machine Limitations

RAM requirements grow as O(n2)

flop count grows asO(n3

)The time for data movement is an important speed bottleneck on modernsystems

An architecture observation...
http://find/


54/55

Disk

FPU CPU

internal bus

cache

system bus

RAM

network

Computer

Data Access Time:

shortest

longest

Stand alone computer

Computer Computer
http://find/


55/55

Operation FLOPSuTv O(n)Au O(n2)AB O(n3)

quiz...

order each in terms ofspeed

order each in terms ofefficiency
http://find/

Documents

Chapter 8 - Solving Linear Systems of Equations